Exact string match in multi line string - awk

I have a string as below, I needed to exactly match if string "abc.properties abc Scanner directory: /xyz/xzy/" is present.
I am using the below awk command.
It considers as successful if even a part of string, say, abc.properties is present. But, my criteria is to check for the exact string i.e. "abc.properties abc Scanner directory: /xyz/xzy/".
Can somebody please help?
echo $s | awk 'BEGIN { FS="\n"; RS="";} /abc.properties abc Scanner directory: \/xyz/xzy\// {print $1}'
String is as below:
spawn ssh -oStrictHostKeyChecking=no -oCheckHostIP=no -o ServerAliveInterval=600 -o ServerAliveCountMax=3 -o ConnectTimeout=30 xyz#123.abc.com Password: Creating directory '/home/xyz'. Authorized uses only. All activity may be monitored and reported. Successful login using -u option. If you want to allow X traffic please use -x option. Usage: /111/11/111/111 [policy] -x [target user] su from xyz to abc [abc#xyz ~]$ /111/111/111/111/111/111.sh /111/11/instance/111/111/111/111/ Verifying JVM setting for domain: /111/11/11/11/1111/111/111// abc.properties abc Scanner directory: /xyz/xzy/ [abc#xyz ~]$ exit 0 logout
The output is as below:
$ echo $s | awk 'BEGIN { FS="\n"; RS="";} /abc.properties abc Scanner directory: \/xyz/xzy\// {print $1}'
spawn ssh -oStrictHostKeyChecking=no -oCheckHostIP=no -o ServerAliveInterval=600 -o ServerAliveCountMax=3 -o ConnectTimeout=30 xyz#123.abc.com Password: Creating directory '/home/xyz'. Authorized uses only. All activity may be monitored and reported. Successful login using -u option. If you want to allow X traffic please use -x option. Usage: /111/11/111/111 [policy] -x [target user] su from xyz to abc [abc#xyz ~]$ /111/111/111/111/111/111.sh /111/11/instance/111/111/111/111/ Verifying JVM setting for domain: /111/11/11/11/1111/111/111// abc.properties abc Scanner directory: /xyz/xzy/ [abc#xyz ~]$ exit 0 logout

Related

How to filter output of a URL

I have a URL and when I send a request by curl, I get a big output.
curl https://www.aparat.com/video/video/embed/videohash/lXhkG/vt/frame -H "Accept: application/json" -s
I get: https://pastebin.mozilla.org/QM6FN8MZ#L
But I just want to get the URL of 720p, I mean just:
https:\/\/caspian1.cdn.asset.aparat.com\/aparat-video\/de54245e862b62249b6b7958c734276547445778-720p.apt?wmsAuthSign=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbiI6IjQ2NDJhYmQ4NGFiN2UzNDJkNGMxZWI3ZTNkMzlmZmQ5IiwiZXhwIjoxNjY5ODA5NzI1LCJpc3MiOiJTYWJhIElkZWEgR1NJRyJ9.havkkhJyXjBt_jHPVv4poEVb65_7tRsLIxO5pCO7tGE
Any idea how to do it?
I'm trying to use grep but I don't know how to remove other things from else 720p URL.
curl https://www.aparat.com/video/video/embed/videohash/lXhkG/vt/frame -H "Accept: application/json" -s | grep -e "720p"
You could go the html-parsing/json-parsing route, e.g.:
curl -s https://www.aparat.com/video/video/embed/videohash/lXhkG/vt/frame |
# Normalize html
xmlstarlet fo -o -H -R 2> /dev/null |
# Extract relevant js bit
xmlstarlet sel -t -v '_:html/_:body/_:div/_:script' 2> /dev/null |
# Extract relevant json
sed -nE '/^ *var +options *= */ { s///; s/;$//p; }' |
# Extract desired url, i.e. the 720p in this case
jq -r '.multiSRC[][] | select( .label == "720p" ) | .src'
I would harness GNU AWK for this following way
wget --quiet -O - https://www.aparat.com/video/video/embed/videohash/lXhkG/vt/frame | awk 'match($0, /http[^"]*720[^"]*/){print substr($0,RSTART,RLENGTH)}'
gives output
https:\/\/caspian1.cdn.asset.aparat.com\/aparat-video\/de54245e862b62249b6b7958c734276547445778-720p.apt?wmsAuthSign=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbiI6IjY1OTcxYTRkNGZiMjkyYjk0NjM0Mjk2ODVkOTc3YjEwIiwiZXhwIjoxNjY5ODIxNDM2LCJpc3MiOiJTYWJhIElkZWEgR1NJRyJ9.NI2_6nwOxLEOxhWghsR2bOqzrXINXqqscbduHpCWwok
Explanation: I use wget with information like progress bar &c turned-off (--quiet) and writing to standard output (-O -) which is piped into awk, which for each line is matching against following regular expression http[^"]*720[^"]* that is http followed by zero-or-more (*) not-quotes followed by 720 followed by zero-or-more non-quotes, if there is match I print substring of line containing that match. match string function sets RSTART and RLENGTH variables, which I use later in substr. Note: this might give false positivie if there are others URL containing 720.
(tested in GNU Wget 1.20.3 and GNU Awk 5.0.1)
Using any awk:
$ cat file | awk 'match($0,/"https?:\\\/\\\/[^"]*-720p\.apt\?[^"]*"/) { print substr($0,RSTART+1,RLENGTH-2) }'
https:\/\/caspian1.asset.aparat.com\/aparat-video\/de54245e862b62249b6b7958c734276547445778-720p.apt?wmsAuthSign=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0b2tlbiI6ImViODhjZDNlYzZhYzk3OTBhZDc3MWJhMzIyNWQ3NmZlIiwiZXhwIjoxNjY5ODE4Mjc5LCJpc3MiOiJTYWJhIElkZWEgR1NJRyJ9.e6do9Ha9EkDS46NZDoHT2dYHSOezu_TbdGAGblfi2tM
The contents of file are what you provided in pastebin, obviously just replace cat file with your curl command.

GNU Parallel -q option causing BCP "unknown option" errors (different string quotes on local vs remote hosts)

Seeing very strange behavior where when when using gnu parallel to distribute export jobs using bcp from mssql-tools. It appears that when using the -q option for parallel, strings are interpreted differently on local host than on remote hosts.
Running only as a loop through files on local host, the bcp processes throws no errors
However, distributing the file exports with parallel, the bcp processes executing on the local host throw
/opt/mssql-tools/bin/bcp: unknown option
errors, while those executing on remote hosts (via a --sshloginfile param) finish successfully. The basic code being run looks like...
# setting some vars to pass
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
TO_SERVER_IP="-S 172.18.54.22"
DB="$dest_db" #TODO: enforce being more careful with this value
TABLE="$tablename" # MUST exist beforehand, case matters
USER=$(tail -n+1 $source_home/mssql-creds.txt | head -1)
PASSWORD=$(tail -n+2 $source_home/mssql-creds.txt | head -1)
DATAFILES="/some/path/to/files/"
TARGET_GLOB="*.tsv"
RECOMMEDED_IMPORT_MODE='-c' # makes a HUGE difference, see https://stackoverflow.com/a/16310219/8236733
DELIMITER="\\\t" # (currently not used) DO NOT use format like "'\t'", nested quotes seem to cause hard-to-catch error, want "\t" literal
....
bcpexport() {
filename=$1
TO_SERVER_ODBCDSN=$2
DB=$3
TABLE=$4 # MUST exist beforehand, case matters
USER=$5
PASSWORD=$6
RECOMMEDED_IMPORT_MODE=$7 # makes a HUGE difference, see https://stackoverflow.com/a/16310219/8236733
DELIMITER=$8 # not currently used
WORKDIR=$9
LOGDIR=${10}
....
/opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE
-t "\t" \
-e ${localfile}.bcperror.log
}
export -f bcpexport
parallelization_pernode=5
parallel -q -j $parallelization_pernode \
--sshloginfile $source_home/parallel-nodes.txt \
--env bcpexport \
bcpexport {} "$TO_SERVER_ODBCDSN" $DB $TABLE $USER $PASSWORD $RECOMMEDED_IMPORT_MODE $DELIMITER $workingdir $logdir \
::: $DATAFILES/$TARGET_GLOB #from hdfs nfs gateway
Looking at the bash interpretation of the processes (by running ps -aux | grep bcp on the hosts that parallelis given in the --sshloginfile) for the remote hosts we see...
/bin/bash -c bcpexport() { ... /opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" $TO_SERVER_ODBCDSN -U $USER -P $PASSWORD -d $DB $RECOMMEDED_IMPORT_MODE; -t "\t" -e ${localfile}.bcperror.log; ...
for the local host, the bash interpretation is...
/bin/bash -c bcpexport() { ... /opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" $TO_SERVER_ODBCDSN -U $USER -P $PASSWORD -d $DB $RECOMMEDED_IMPORT_MODE; -t "\t" -e ${localfile}.bcperror.log; ...
that is, they look the same.
My current thought is that the "\t" in the bcp command is being interpreted in a problematic way. Debugging parallel without vs with the -q option we see...
$ parallel -j 5 --sshloginfile ./parallel-nodes.txt echo "Number {}: Running on \`hostname\`: \t" ::: 1 2 3 4 5
Number 4: Running on HW04.ucera.local: t
Number 1: Running on HW04.ucera.local: t
Number 2: Running on HW03.ucera.local: t
Number 5: Running on HW03.ucera.local: t
Number 3: Running on HW02.ucera.local: t
$ parallel -q -j 5 --sshloginfile ./parallel-nodes.txt echo "Number {}: Running on \`hostname\`: \t" ::: 1 2 3 4 5
Number 1: Running on `hostname`:
Number 4: Running on `hostname`:
Number 3: Running on `hostname`: \t
Number 2: Running on `hostname`: \t
Number 5: Running on `hostname`: \t
The bcp command needs the "\t" literal not the "t" literal (and I suspect several other similar string corruptions (also I do believe that \t is the default for bcp anyway, but this is just an example and want to keep \t for code clarity)), but not sure how to get this for both local and remote nodes or even why this behavior differs by remote vs local.
Basically, need the the strings to be exactly the same for both local and remote hosts even if strings have spaces or escape characters in them (note, I think this used to not be the case (have older script on other machines that don't have this problem))
Not sure if this is counts more as a parallel problem or a bcp problem (currently thinking something is going wrong with the -q option in parallel, but not sure). Anyone have any debugging suggestions or fixes? Ideas of what could be happening?
Firstly, the reason why hostname is not expanded is due to -q. It quotes the ` so that it does not expand.
Secondly, I think what you see is the different behaviours in built-in echo and /bin/echo. Built-in echo depends on the shell. Here I compare echo \\\\t in different shells:
$ parallel --onall --tag -S sh#lo,bash#lo,csh#lo,tcsh#lo,ksh#lo,zsh#lo echo \\\\t ::: a
bash#lo \t a
tcsh#lo a
sh#lo a
ksh#lo \t a
zsh#lo a
csh#lo \t a
That does not, however, get you closer to a solution. If I were you I would use env_parallel to copy the environment variables. And if the login shell on the remote systems are not the same as your shell, then set PARALLEL_SHELL to force using that shell.
So:
#!/bin/bash
env_parallel --session
# setting some vars to pass
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
:
:
PARALLEL_SHELL=bash env_parallel -q -j $parallelization_pernode ...
(no need to use neither --env nor 'export -f' when using 'env_parallel --session')
# Cleanup (not needed if this is the last line in the script)
env_parallel --end-session

How to enable sshpass output to console

Using scp and interactively entering the password the file copy progress is sent to the console but there is no console output when using sshpass in a script to scp files.
$ sshpass -p [password] scp [file] root#[ip]:/[dir]
It seems sshpass is suppressing or hiding the console output of scp. Is there a way to enable the sshpass scp output to console?
After
sudo apt-get install expect
the file send-files.exp works as desired:
#!/usr/bin/expect -f
spawn scp -r $FILES $DEST
match_max 100000
expect "*?assword:*"
send -- "12345\r"
expect eof
Not exactly what was desired, but better than silence:
SSHPASS="12345" sshpass -e scp -v -r $FILES $DEST 2>&1 | grep -v debug1
Note that -e is considered a bit safer than -p.
Output:
Executing: program /usr/bin/ssh host servername, user username, command scp -v -t /src/path/dst_file.txt
OpenSSH_6.6.1, OpenSSL 1.0.1i-fips 6 Aug 2014
Authenticated to servername ([10.11.12.13]:22).
Sending file modes: C0600 590493 src_file.txt
Sink: C0600 590493 src_file.txt
Transferred: sent 594696, received 2600 bytes, in 0.1 seconds
Bytes per second: sent 8920671.8, received 39001.0
In this way:
output=$(sshpass -p $PASSWD scp -v $filename root#192.168.8.1:/root 2>&1)
echo "Output = $output"
you redirect the console output in variable output.
Or, if you only want to see the console output of scp command, you should add only -v command in your ssh pass cmd:
sshpass -p $PASSWD scp -v $filename root#192.168.8.1:/root

Expect script SSH not successful

I have made two scripts:
This one fetches IP address & Hostnames:
#!/bin/bash
for i in `cat ~/script/hosts.txt`
do HOSTNAME=`echo $i|awk -F: '{print $1}'`
IP=`echo $i|awk -F: '{print $2}'`
TIMESTAMP=`date +%Y-%m-%d`
~/script/expect.sh $HOSTNAME $IP
done
This one does SSH into the devices:
#!/usr/bin/expect
set timeout 20
set HOSTNAME [lindex $argv 0]
set IP [lindex $argv 1]
exp_internal 1
spawn ssh -o StrictHostKeyChecking=no root#$IP
exit
I want to make a script to bakcup multiple device configurations.
Problem is that SSH is failing due to following errors:
$ ./main.sh
spawn ssh -o StrictHostKeyChecking=no root#10.102.82.235
: Name or service not knownname 10.102.82.235
spawn ssh -o StrictHostKeyChecking=no root#10.102.82.239
: Name or service not knownname 10.102.82.239
When I debug, I see the following error
spawn id exp4 sent <ssh: Could not resolve hostname 10.102.82.235\r\r: Name or
service not known\r\r\n>
: Name or service not knownname 10.102.82.235
I think the issue is due to these characters: "\r\r", "\r\r\n"
Is there any way I can filter these out?
Not an answer, but your shell script can use much improvement:
#!/bin/bash
while IFS=: read -r host ip; do
timestamp=$(date +%Y-%m-%d %T)
~/script/expect.sh "$host" "$ip"
done < ~/script/hosts.txt
Notes:
don't use for line in `cat file` to read the lines of a file -- a for loop reads words from a file
use $(...) for command substitution, not `...` -- improved readability, and easy to nest
don't use UPPERCASE_VARIABLES -- those should be reserved for the shell's use.
your (unused) TIMESTAMP variable actually contains a date, no time.
quote your "$variables" unless you can explain why you want them unquoted.

How do you htdigest 400 user accounts?

How do you generate user accounts for 400 users to do a load testing?
Htdigest forces you to type in a password each time, I have tried dos pipes like
echo password > htdigest -c realm username%1
htdigest -c realm username%1 < password.txt
but it is not working...
You can also check out the python script that trac distributes on their website for htdigest passwords, you can then automate it:
Generating htdigest passwords without Apache
They also suggest something along these lines will work:
It is possible to use md5sum utility to generate digest-password file using such method:
$ printf "${user}:trac:${password}" | md5sum - >>user.htdigest
and manually delete " -" from the end and add "${user}:trac:" to the start of line from 'to-file'.
I have tested this on FreeBSD, not sure if this will work on Linux or Windows, so you may need to modify it a little:
(echo -n "user:realm:" && echo -n "user:realm:testing" | md5) > outfile
outfile contains:
user:realm:84af20dd88a2456d3bf6431fe8a59d16
Same thing with htdigest:
htdigest -c outfile2 realm user
output in outfile2
user:realm:84af20dd88a2456d3bf6431fe8a59d16
They are both the same, thereby proving correctness of the command line implementation!
(Aside: On unix/linux the first one should be:
echo password | htdigest -c realm username$1
)
As htdigest doesn't have any nice way to pass the password in, I would use expect to automate the process.
An example from http://www.seanodonnell.com/code/?id=21:
#!/usr/bin/expect
#########################################
#$ file: htpasswd.sh
#$ desc: Automated htpasswd shell script
#########################################
#$
#$ usage example:
#$
#$ ./htpasswd.sh passwdpath username userpass
#$
######################################
set htpasswdpath [lindex $argv 0]
set username [lindex $argv 1]
set userpass [lindex $argv 2]
# spawn the htpasswd command process
spawn htpasswd $htpasswdpath $username
# Automate the 'New password' Procedure
expect "New password:"
send "$userpass\r"
expect "Re-type new password:"
send "$userpass\r"
It's left as an exercise to the user to convert this for Windows if required.
Here is a script that will read in a list of user names, generate a random password for each, and output them to both an htdigest file, and a plain text file. It has been tested on Linux, but may need to be modified for other systems. In particular, md5sum may be md5, and head does always accept the -c flag.
#!/bin/bash
# auth realm for digest auth
AUTH_REALM=MyRealm
# file locations
# a file containing a list of user names,
# one name per line, e.g.,
# $ cat users.txt
# joe
# curly
# larry
USER_FILE=users.txt
# htdigest file, needs to exist
HTDIGEST_FILE=passwd.htdigest
# insecure password file
PASSWD_FILE=passwd.txt
# read the names from the user file
while read username
do
# generate a pseudo-random password
rand_pw=`< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c8`
# hash the username, realm, and password
htdigest_hash=`printf $username:$AUTH_REALM:$rand_pw | md5sum -`
# build an htdigest appropriate line, and tack it onto the file
echo "$username:$AUTH_REALM:${htdigest_hash:0:32}" >> $HTDIGEST_FILE
# put the username and password in plain text
# clearly, this is terribly insecure, but good for
# testing and importing
echo "$username:$rand_pw" >> $PASSWD_FILE
done < $USER_FILE
This is what the input and results look like, first the user names file:
$ cat users.txt
joe
curly
larry
Running the script:
$ ./load_users.bash
The resulting htdigest file:
$ cat passwd.htdigest
joe:MyRealm:2603a6c581f336f2874dbdd253aee780
curly:MyRealm:fd3f9d87bba654439d5ba1f32c0286a8
larry:MyRealm:c1c3c0dc50a9b97e9f7ee582e3fce892
And the plain text file:
$ cat passwd.txt
joe:aLnqnrv0
curly:3xWxHKmv
larry:7v7m6mXY