I set up a variable in the shell:
-bash-3.00$ echo $source_repl
source ../setup_simple.tcl
I then tried to replace a line in a file that started with the string "package require IxLoad" with the variable string (noting that double quotes are the way to get sed to use variable substitution). First I tried with direct substitution (no escaping the $ in the variable):
-bash-3.00$ sed -e "s/package require IxLoad.*/$source_repl/g" smtp_tput191Mb.tcl > tmpwatch.tcl
sed: -e expression #1, char 38: unknown option to `s'
-bash-3.00$
So I thought that escaping the $ would solve the problem but as you can see the line is then replaces by the literal string "$source_repl" rather than the variable stored there:
-bash-3.00$ sed -e "s/package require IxLoad.*/\$source_repl/g" smtp_tput191Mb.tcl > tmpwatch.tcl
-bash-3.00$ diff smtp_tput191Mb.tcl tmpwatch.tcl
11c11
< package require IxLoad
---
> $source_repl
-bash-3.00$
I looked up many sites on how to do variable substitution in sed and they all seem to indicate that the above should work. What am I missing? Is there something in the actual variable that's causing this?
A
There is a / in your variable. Use # as pattern separator and this will go.
sed -e "s#package require IxLoad.*#$source_repl#g" smtp_tput191Mb.tcl > tmpwatch.tcl
I don't know how to work this issue around in general (changing the separators is easy, but you never know what can be in the variable), but if you can use perl instead of sed, you could load the environment variable form inside perl to avoid the escaping issue:
perl -pe 's/package require IxLoad.*/$ENV{source_repl}/g' smtp_tput191Mb.tcl > tmpwatch.tcl
Sorry for wasting your time folks. I found the problem. I needed to escape the dots and slashes in the actual variable in order to replace it with the sed statement:
-bash-3.00$ echo $source_repl
source \.\.\/setup_simple\.tcl
-bash-3.00$ sed -e "s/package require IxLoad.*/$source_repl/g" smtp_tput191Mb.tcl > tmpwatch.tcl
-bash-3.00$ diff smtp_tput191Mb.tcl tmpwatch.tcl 11c11
< package require IxLoad
---
> source ../setup_simple.tcl
-bash-3.00$
Thanks
Related
I have ./cpptest.sh to which I am passing a command line parameter
For e.g.
$./testcps.sh /srv/repository/Software/Wind_1.0.2/
The above command line parameter, is stored in variable $1
when I echo $1, the output is correct (the path)
Actual issue...
There is another file let's say abc.properties file. In this file there is a key-value field something like location.1=stg_area.
I want to replace the 'stg_area' with the value stored in $1 (the path) so that the substitution looks like location.1=/srv/repository/Software/Wind_1.0.2/
Now, to achieve this, I am tried all option below with sed and none worked
sed -i "s/stg_area/$1/" /srv/ppc/abc.properties //output is sed: -e expression #1, char 17: unknown option to `s'
sed -i 's/stg_area/'"$1'"/' /srv/ppc/abc.properties //output is sed: -e expression #1, char 18: unknown option to `s'
sed -i s/stg_area/$1/ /srv/ppc/abc.properties //output is sed: -e expression #1, char 17: unknown option to `s'
I think I have tried all possible ways... Any answer on this is appreciated. Thanks in advance.
You know that sed is using / as a special separator in the command s/pattern/replacement/, right? You've used it yourself in the question.
So obviously there's a problem when you have a replacement string containing /, like yours does.
As the documentation says:
The / characters may be uniformly replaced by any other single character within any given s command. The / character (or whatever other character is used in its stead) can appear in the regexp or replacement only if it is preceded by a \ character.
So the two available solutions are:
use a different separator for the s command, such as
s#stg_area#$1#
(although you still need to check there are no # characters in the replacement string)
sanitize the replacement string so it doesn't contain any special characters (either /, or sequences like \1, or anything else sed treats as special), for example by escaping them with \
sanitized=$(sed 's#/#\\/#g' <<< $1)
(and then used $sanitized instead of $1 your sed script)
In the following file I want to replace all the ; by , with the exception that, when there is a string (delimited with two "), it should not replace the ; inside it.
Example:
Input
A;B;C;D
5cc0714b9b69581f14f6427f;5cc0714b9b69581f14f6428e;1;"5cc0714b9b69581f14f6427f;16a4fba8d13";xpto;
5cc0723b9b69581f14f64285;5cc0723b9b69581f14f64294;2;"5cc0723b9b69581f14f64285;16a4fbe3855";xpto;
5cc072579b69581f14f6428a;5cc072579b69581f14f64299;3;"5cc072579b69581f14f6428a;16a4fbea632";xpto;
output
A,B,C,D
5cc0714b9b69581f14f6427f,5cc0714b9b69581f14f6428e,1,"5cc0714b9b69581f14f6427f;16a4fba8d13",xpto,
5cc0723b9b69581f14f64285,5cc0723b9b69581f14f64294,2,"5cc0723b9b69581f14f64285;16a4fbe3855",xpto,
5cc072579b69581f14f6428a,5cc072579b69581f14f64299,3,"5cc072579b69581f14f6428a;16a4fbea632",xpto,
For sed I have: sed 's/;/,/g' input.txt > output.txt but this would replace everything.
The regex for the " delimited string: \".*;.*\" .
(A regex for hexadecimal would be better -- something like: [0-9a-fA-F]+)
My problem is combining it all to make a grep -o / sed that replaces everything except for that pattern.
The file size is in the order of two digit Gb (max 99Gb), so performance is important. Relevant.
Any ideas are appreciated.
sed is for doing simple s/old/new on individual strings. grep is for doing g/re/p. You're not trying to do either of those tasks so you shouldn't be considering either of those tools. That leaves the other standard UNIX tool for manipulating text - awk.
You have a ;-separated CSV that you want to make ,-separated. That's simply:
$ awk -v FPAT='[^;]*|"[^"]+"' -v OFS=',' '{$1=$1}1' file
A,B,C,D
5cc0714b9b69581f14f6427f,5cc0714b9b69581f14f6428e,1,"5cc0714b9b69581f14f6427f;16a4fba8d13",xpto,
5cc0723b9b69581f14f64285,5cc0723b9b69581f14f64294,2,"5cc0723b9b69581f14f64285;16a4fbe3855",xpto,
5cc072579b69581f14f6428a,5cc072579b69581f14f64299,3,"5cc072579b69581f14f6428a;16a4fbea632",xpto,
The above uses GNU awk for FPAT. See What's the most robust way to efficiently parse CSV using awk? for more details on parsing CSVs with awk.
If I get correctly your requirements, one option would be to make a three pass thing.
From your comment about hex, I'll consider nothing like # will come in the input so you can do (using GNU sed) :
sed -E 's/("[^"]+);([^"]+")/\1#\2/g' original > transformed
sed -i 's/;/,/g' transformed
sed -i 's/#/;/g' transformed
The idea being to replace the ; when within quotes by something else and write it to a new file, then replace all ; by , and then set back the ; in place within the same file (-i flag of sed).
The three pass can be combined in a single command with
sed -E 's/("[^"]+);([^"]+")/\1#\2/g;s/;/,/g;s/#/;/g' original > transformed
That said, there's probably a bunch of csv parser witch already handle quoted fields that you can probably use in the final use case as I bet this is just an intermediary step for something else later in the chain.
From Ed Morton's comment: if you do it in one pass, you can use \n as replacement separator as there can't be a newline in the text considered line by line.
This might work for you (GNU sed):
sed -E ':a;s/^([^"]*("[^"]*"[^"]*)*"[^";]*);/\1\n/;ta;y/;/,/;y/\n/;/' file
Replace ;'s inside double quotes with newlines, transpose ;'s to ,'s and then transpose newlines to ;'s.
I have the following value from a variable .
manager&org.apache.catalina.filters.CSRF_NONCE=314C54E5671790D592A37C2C4A6B9AAF
I need to modify the above variable to remove & from it. SO , the variable should like this
manager&;org.apache.catalina.filters.CSRF_NONCE=314C54E5671790D592A37C2C4A6B9AAF
Please suggest
I put your variable in a file and was able to do what you wanted with sed. The trick to make sure you do not remove any other references to just amp is to include the & as part of the substitution.
$ cat /tmp/file
manager&org.apache.catalina.filters.CSRF_NONCE=314C54E5671790D592A37C2C4A6B9AAF
$ cat /tmp/file | sed 's/\&/\&/g'
manager&;org.apache.catalina.filters.CSRF_NONCE=314C54E5671790D592A37C2C4A6B9AAF
sed 's/&/\&;/g' <<<"yourString"
The above line should help.
example:
kent$ sed 's/&/\&;/g'<<< "foo&bar&blah"
foo&;bar&;blah
In bash, you can use Parameter Expansion - Pattern substitution to remove a substring:
VAR='manager&org.apache.catalina.filters.CSRF_NONCE=314C54E5671790D592A37C2C4A6B9AAF'
echo ${VAR/&}
I have a lot of text files in which I would like to find the word 'CASE' and replace it with the related filename.
I tried
find . -type f | while read file
do
awk '{gsub(/CASE/,print "FILENAME",$0)}' $file >$file.$$
mv $file.$$ >$file
done
but I got the following error
awk: syntax error at source line 1 context is >>> {gsub(/CASE/,print <<< "CASE",$0)}
awk: illegal statement at source line 1
I also tried
for i in $(ls *);
do
awk '{gsub(/CASE/,${i},$0)}' ${i} > file.txt;
done
getting an empty output and
awk: syntax error at source line 1 context is >>> {gsub(/CASE/,${ <<<
awk: illegal statement at source line 1
Why awk? sed is what you want:
while read -r file; do
sed -i "s/CASE/${file##*/}/g" "$file"
done < <( find . -type f )
or
while read -r file; do
sed -i.bak "s/CASE/${file##*/}/g" "$file"
done < <( find . -type f )
To create a backup of the original.
You didn't post any sample input and expected output so this is a guess but maybe this is what you want:
find . -type f |
while IFS= read -r file
do
awk '{gsub(/CASE/,FILENAME)} 1' "$file" > "${file}.$$" &&
mv "${file}.$$" "$file"
done
Every change I made to the shell code is important so if you don't understand why I changed any part of it, ask the question.
btw if after making the changes you are still getting the error message:
awk: syntax error at source line 1
awk: illegal statement at source line 1
then you are using old, broken awk (/usr/bin/awk on Solaris). Never use that awk. On Solaris use /usr/xpg4/bin/awk (or nawk if you must).
Caveats: the above will fail if your file name contains newlines or ampersands (&) or escaped digits (e.g. \1). See Is it possible to escape regex metacharacters reliably with sed for details. If any of that is a problem, post some representative sample input and expected output.
print in that first script is the error.
The second argument to gsub is the replacement string not a command.
You want just FILENAME. (Note not "FILENAME" that's a literal string. FILENAME the variable.)
find . -type f -print0 | while IFS= read -d '' file
do
awk '{gsub(/CASE/,FILENAME,$0)} 7' "$file" >"$file.$$"
mv "$file.$$" "$file"
done
Note that I quoted all your variables and fixed your find | read pipeline to work correctly for files with odd characters in the names (see Bash FAQ 001 for more about that). I also fixed the erroneous > in the mv command.
See the answers on this question for how to properly escape the original filename to make it safe to use in the replacement portion of gsub.
Also note that recent (4.1+ I believe) versions of awk have the -i inplace argument.
To fix the second script you need to add the quotes you removed from the first script.
for i in *; do awk '{gsub(/CASE/,"'"${i}"'",$0)}' "${i}" > file.txt; done
Note that I got rid of the worse than useless use of ls (worse than useless because it actively breaks files with spaces or shell metacharacters in the their names (see Parsing ls for more on that).
That command though is somewhat ugly and unsafe for filenames with various characters in them and would be better written as the following though:
for i in *; do awk -v fname="$i" '{gsub(/CASE/,fname,$0)}' "${i}" > file.txt; done
since that will work with filenames with double quotes/etc. in their names correctly whereas the direct variable expansion version will not.
That being said the corrected first script is the right answer.
I have a bash script that I use to setup a simple php script on my server. I am stuck with how to correctly change a variable with sed with the script. Here is what I have tried:
echo "Enter Portal Password:"
read PORTPASS;
sed -i 's/$ppass =".*"/$ppass ="$PORTPASS"/' includes/config.php
The above changes the variable in the config file but it only changes it to $PORTPASS it does not change it to what I input in the script.
I also tried this and it does change the $PORTPASS correctly, but it remove the " " around the variable in the file.
sed -i 's/$ppass ='".*"'/$ppass ='"$PORTPASS"';/' includes/config.php
Here is what I'm trying to change in the conf.php file: $ppass ="password";
Try:
sed -i "s/\$ppass =\".*\"/\$ppass =\"$PORTPASS\"/" includes/config.php
You have to use double quotes (") around the command so that the shell will evaluate $PORTPASS before passing it to sed, so then you have to "escape" all of the double quotes within the command.