I have a file in which i have to find a String and replace it. It giving me error when i am using sed because the replace string contains "/".
replace_string=6W4kngjd3c7oOShnG7iWYQpZVzr4S88G20fujmP7cdM1m5Gw550WfMD38DA4g6O4qxUIJJwt2OtLTRmh7vWz+AWQVmIMajk3OylEfR/X+afrD6YOeGLYHU6Ef4DYv/3x
sed -i -e 's|string|'$replace_string'|g' $FILEPATH
This is the error which its giving -
sed: -e expression #1, char 38: unterminated `s' command
Is there any other method other than sed or any other way i can use sed?
sed doesn't understand literal strings (see Is it possible to escape regex metacharacters reliably with sed), only regexps and backreference-enabled replacements that can't contain whatever delimiters you use. If you want to replace a string then use a tool that understands strings such as awk, e.g. this will work for any characters in your strings (code would need tweaked to handle newlines within old or multiple substitutions on a single line):
old='string1' new='string2' awk '
BEGIN {
old = ENVIRON["old"]
new = ENVIRON["new"]
lgth = length(old)
}
s = index($0,old) { $0 = substr($0,1,s-1) new substr($0,s+lgth) }
{ print }
' file
ripgrep is another tool that allows to search and replace literally. The substitution capability is limited compared to sed (for ex: it always replaces all occurences like g flag)
$ cat ip.txt
2.3/[4]*6+[4]*2
foo
5.3-[4]*9
$ rg -N --passthru -F '[4]*' -r '\&/' ip.txt
2.3/\&/6+\&/2
foo
5.3-\&/9
For inplace editing:
rg --passthru -F 'old' -r 'new' ip.txt > tmp && mv tmp ip.txt
With shell variables:
rg --passthru -F "$old" -r "$new" ip.txt > tmp && mv tmp ip.txt
-N flag not required when redirecting output of rg
Related
I've a file inside which I have a placeholder text for a password. I'm now trying to find and replace the placeholder text with the actual password. The text looks like below:
USER_GIVEN_PASSWORD=<my_password>
Let's say my password is: ABC&12345
I'm using the below command inside a script to replace this:
sed -i "s/<my_password>/$1/g" file.txt
I pass the input to my script as below:
sh password_replace.sh ABC&12345
My expected output is:
USER_GIVEN_PASSWORD=ABC&12345
But I'm getting the below output:
USER_GIVEN_PASSWORD=ABC<my_password>12345
Clearly, I'm doing something wrong with the & symbol present in my password. So, when I tried with escaping & in my input as follows, it actually works:
sh password_replace.sh ABC'\&'12345
But the problem is I should not adjust the input parameter to pass an escape character because the password won't be manually typed. It will automatically come from something like the Azure key vault as the input to my script.
So, I need to make the sed command itself handle the incoming special characters.
Can someone please help me achieve this?
Sed doesn't understand literal strings, just regexps and backreference-enabled replacement text. See Is it possible to escape regex metacharacters reliably with sed for the lengths you have to go to to try to get sed to behave as if you were using literal strings.
Just use the literal string functions in awk instead:
$ new='ABC&12\345' awk '
BEGIN { old="<my_password>"; lgth=length(old); new=ENVIRON["new"] }
s = index($0,old) { $0 = substr($0,1,s-1) new substr($0,s+lgth) }
1' file
USER_GIVEN_PASSWORD=ABC&12\345
That will work for any character, no need to do any escaping.
In your shell script where you're trying replace with $1 you'd just do:
new="$1" awk 'script' file
If you're using GNU awk you can use -i inplace just like you can use -i with GNU sed:
new="$1" awk -i inplace 'script' file
or with any awk:
tmp=$(mktemp)
new="$1" awk 'script' file > "$tmp" && mv "$tmp" file
It's important for new=... to be at the start of the same line as the awk command starts on so ENVIRON[] can access it without it having to be exported.
As others have commented, & is a reserved word in sed and will need to be "escaped" in order to be used in substitute text. An alternative approach would be to use awk (GNU awk in this case):
awk -v pass="$pass" -v confpass="$confpass" -v schepass1="$schepass1" -v schepass2="$schpass2" '/USER_GIVEN_PASS/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)pass""substr($0,RSTART+RLENGTH) } /CONFIRM_USER_GIVEN_PASSWORD/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)confpass""substr($0,RSTART+RLENGTH) } /SCHEMA1_PASSWORD/ { match($0,"<db_password>");$0=substr($0,1,RSTART-1)schpass1""substr($0,RSTART+RLENGTH) } /SCHEMA2_PASSWORD/ { match($0,"<db_password>");$0=substr($0,1,RSTART-1)schpass2""substr($0,RSTART+RLENGTH) }1' file
Pass the password to awk as a variable "pass". When the line containing "USER_GIVEN_PASS" is encountered, match the position of the "holder" using awk's match function. Set the line ($0) equal to the string up to the place holder, pass, then the rest of the string.
If you have later versions of GNU awk, you can use -i to commit the changes back to the file once you are happy.
Otherwise:
awk -v pass="$pass" '/USER_GIVEN_PASS/ { match($0,"<my_password>");$0=substr($0,1,RSTART-1)pass""substr($0,RSTART+RLENGTH) }1' file > tmpfile && mv -f tmpfile file
I have the following lines in a text file:
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:FOO.${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:BAR.${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:BAZ.${TAG_NAME}
I want to write a sed or awk command that finds all occurrences of FOO., BAR. and BAZ. in the above lines in a file and deletes these occurrences so that the result looks like this in the end:
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
Sed one:
sed 's/\b\(FOO\|BAR\|BAZ\)\.//' input_file
When you want to replace to first occurance of :something. in every line, test
sed 's/:[^.]*[.]/:/' inputfile
When this works and you want it replaced in the file without making a backup, you can use the option -i:
sed -i 's/:[^.]*[.]/:/' inputfile
When ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME} might have a :, the pattern is matched at the wrong place. When you are sure that the ${TAG_NAME} is without :, use
sed -r 's/:[^.]*[.]([^:]*)$/:\1/' inputfile
Edit: After the comment of #potong, I replaced '.' with ':' in the replacement strings. I kept the wrong character.
Use one of these Perl one-liners:
# Remove { FOO, BAR or BAZ } and the following '.' :
perl -pe 's/(FOO|BAR|BAZ)[.]//' in_file > out_file
# Remove *anything* between the last ':' (exclusive) and '.' (inclusive) :
perl -pe 's/\A(.+:)[^.]+[.]/$1/' in_file > out_file
The Perl one-liners use these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
The regex uses:
\A : beginning of the line,
(.+:) : capture into variable $1 everything between the first character and the first semicolon,
[^.]+[.] : 1 or more occurrence of any character other than '.', followed by '.'. It is surrounded by brackets to match a literal dot: [.]
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlre: Perl regular expressions (regexes)
perldoc perlre: Perl regular expressions (regexes): Quantifiers; Character Classes and other Special Escapes; Assertions; Capture groups
In awk the variable FS containing a regex can help.
$ cat file
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:FOO.${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:BAR.${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:BAZ.${TAG_NAME}
In the input, we can see that the 3 words, each one with 3 characters (FOO, BAR, BAZ) and followed by dot. We can make a regex for the match inside the FS separator...
FS='[A-Z]{3}\\.'
In the awk manual we can read that "the value of FS may be a string containing any regular expression. In this case, each match in the record for the regular expression separates fields."
https://www.gnu.org/software/gawk/manual/html_node/Regexp-Field-Splitting.html
So we have
awk -v FS='[A-Z]{3}\\.' '{print $1 $2}' file
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
..without comma: $1 $2
The same result gives the regex with the 3 specific words:
$ awk -v FS='(FOO|BAR|BAZ)\\.' '{print $1 $2}' file
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
docker pull ${DOCKER_REGISTRY}/{REPOSITORY}/{IMAGE_NAME}:${TAG_NAME}
--
I'm trying to convert text by replacing several identical letters (more than 3) with several characters (two *).
My input:
ffffOOOOuuuurrrr
fffffiiiiivvvvveeeee
What should I get:
**OOOO****
********
My test command is:
awk '{gsub(/[a-z]{4}/,"*"); print}' textfile
I don't understand how to transform {4} into 'more than 3'.
Also how to print * two times (like multiply it).
I'm also sure that the condition 'more than three' will convert input into:
**OOOO**
**
Is there any way to avoid this (replace a sequence of identical letters)?
Or it is not possible to fit in one small command.
POSIX awk or sed don't support back-references. You will need to use gnu-sed or perl:
sed -E 's/([a-z])\1{3,}/**/g' file
**OOOO****
********
or using perl:
perl -pe 's/([a-z])\1{3,}/**/g' file
RegEx Details:
([a-z]): Match [a-z] and capture in group #1
\1: Back-reference of the letter captured in group #1
{3,}: Repeat 3 or more times
You mentioned sed as an option in the tags:
echo "fffffiiiiivvvvveeeee" | sed 's/\([A-Za-z]\)\1\1\1\+/\1/g'
five
echo "fffffiiiiivvveeeee" | sed 's/\([A-Za-z]\)\1\1\1\+/\1/g'
fivvve
Here's how to do it with any awk assuming a locale where lower case letters are a-z = ASCII 97-122:
$ cat tst.awk
{
for (i=97; i<122; i++) {
gsub(sprintf("%c{3,}",i),"**")
}
print
}
$ awk -f tst.awk file
**OOOO****
********
otherwise with GNU awk for the ord() function:
$ cat tst.awk
#load "ordchr"
{
for (i=ord("a"); i<=ord("z"); i++) {
gsub(sprintf("%c{3,}",i),"**")
}
print
}
$ awk -f tst.awk file
**OOOO****
********
or iyou can use a different numeric loop range or split("abc...z",...) or whatever else to get the loop but the point is - you need to loop on each character.
I have a file named mapstring. Because of [ string in my patterns my script is not working. Please help me find a solution to this.
Content of mapstring
BC1 bc1
BC2 bc2
BAD_BIT[0] badl0
BAD_BIT[1] badlleftnr
I am working with following script to replace pattern in file testfile
Content of script
foreach cel (`cat mapstring |awk '{print $1}'`)
echo $cel
grep -wq $cel testfile
if( $status == 0 ) then
set var2 = `grep -w $cel rajeshmap |awk '{print $2}'`
sed -i "s% ${cel} % ${var2} %g" testfile
endif
end
Content of testfile
rajesh jain BAD_BIT[0] 1234 BAD_BIT[1000]
jain rajesh DA[0] snps
raj jain CLK stm
That's because square brackets are reserved in sed's basic regex syntax.
You'll have to escape them (and any other special characters in fact) using backslashes (i.e. \[) before using them later in your script; this can itself be done with sed, e.g.:
sed -re 's/(\[|\])/\\\1/g'
(note that using extended regexes in sed (-r) can make this easier).
Your script is rather inefficient anyhow. You can simply get rid of csh entirely (along with the useless cat and the other stylistic problems), and do this with two connected sed scripts.
sed 's/[][*\\%]/\\&/g;s/\([^ ]*\) *\(.*\)/s%\1%\2%g/' mapstring |
sed -i -f - testfile
This is assuming your sed can accept a script on standard input (-f -) and that your sed dialect does not understand any additional special characters which need to be escaped.
#!/bin/ksh
# or sh
sed 's/[[\\$^&.+*]/\\&/g' mapstring | while read -r OldCel NewCel
do
echo ${OldCel}
sed -i "/${OldCel}/ {
s/.*/ & /;s% ${OldCel} % ${NewCel} %g;s/.\\(.*\\)./\\1/
}" testfile
done
pre escape your cel values for a sed manipulation (you could add other special char if occuring and depending directive given to sed like {( )
try something like this (cannot test, no GNU sed available here)
From the good remarq of #tripleee, this need to be another shell than the one used in the request, script adapted for this
I did search and found how to replace each occurrence of a string in files. Besides that I want to add one line to a file only at the first occurrence of the string.
I know this
grep -rl 'windows' ./ | xargs sed -i 's/windows/linux/g'
will replace each occurrence of string. So how do I add a line to that file at first match of the string? Can any have an idea how to do that? Appreciate your time.
Edited :
Exaple : replace xxx with TTT in file, add a line at starting of file for first match.
Input : file1, file2.
file1
abc xxx pp
xxxy rrr
aaaaaaaaaaaddddd
file2
aaaaaaaaaaaddddd
Output
file1
#ADD LINE HERE FOR FIRST MATCH DONT ADD FOR REST OF MATCHES
abc TTT pp
TTTy rrr
aaaaaaaaaaaddddd
file2
aaaaaaaaaaaddddd
Cribbing from the answers to this question.
Something like this would seem to work:
sed -e '0,/windows/{s/windows/linux/; p; T e; a \new line
;:e;d}; s/windows/linux/g'
From start of the file to the first match of /windows/ do:
replace windows with linux
print the line
if s/windows/linux/ did not replace anything jump to label e
add the line new line
create label e
delete the current pattern space, read the next line and start processing again
Alternatively:
awk '{s=$0; gsub(/windows/, "linux")} 7; (s ~ /windows/) && !w {w=1; print "new line"}' file
save the line in s
replace windows with linux
print the line (7 is true and any true pattern runs the default action of {print})
if the original line contained windows and w is false (variables are empty strings by default and empty strings are false-y in awk)
set w to 1 (truth-y value)
add the new line
If I understand you correctly, all you need is:
find . -type f -print |
while IFS= read -r file; do
awk 'gsub(/windows/,"unix"){if (!f) $0 = $0 ORS "an added line"; f=1} 1' "$file" > tmp &&
mv tmp "$file"
done
Note that the above, like sed and grep would, is working with REs, not strings. To use strings would require the use of index() and substr() in awk, is not possible with sed, and with grep requires an extra flag.
To add a leading line to the file if a change is made using gNU awk for multi-char RS (and we may as well do sed-like inplace editing since we're using gawk):
find . -type f -print |
while IFS= read -r file; do
gawk -i inplace -v RS='^$' -v ORS= 'gsub(/windows/,"unix"){print "an added line"} 1' "$file"
done