Using PowerShell variable for example in FTP.exe - variables

How do you use powershell variables when you run program in it? Is it simply not possible?
Example:
$url = "test.com"
ftp
ftp> open $url
Unknown host $url.

The FTP command will read commands from a file, so you can use Powershell to create a file containing the required commands and then tell the command to read from the file. It is best to create a unique name for the temporary file and remember to delete it when you are finished:
PS C:\> $url = "test.com"
PS C:\> $filename = "foo.bar"
PS C:\> $tempFile = [io.path]::GetTempFileName()
PS C:\> #"
>> open $url
>> get $filename
>> quit
>> "# >>$tempFile
>>
PS C:\> cat $tempFile
open test.com
get foo.bar
quit
PS C:\> ftp -s:$tempFile
PS C:\> remove-item $tempFile
A multi-line string #"..."# is a good way to create a temporary file for external commands because it lets you write the commands in a natural way.
Other commands may not accept input from a file. In that case it is worth checking whether you can specify all of the needed commands on the command line. Many non-Powershell commands will actually accept multi-line arguments even though when run from cmd.exe there would be no way to pass such arguments. For example:
PS C:\python34> python -c #"
>> for i in range(10):
>> print(i)
>> "#
>>
0
1
2
3
4
5
6
7
8
9
PS C:\python34>

Related

Pass arguments for SQL statement in a shell file from another shell file through ssh command

I am passing command line arguments to a shell file i.e assignRole.sh which contains an SQL command which will use these arguments like below
ssh -o StrictHostKeyChecking=no -T $key < /oracle/oracle_user/makhshif/./assignRole.sh name open_mode >> /oracle/oracle_user/dftest.txt
This gives me error and does not accept arguments of name and open_mode and gives error, but if I execute the statement outside of ssh command like:
/oracle/oracle_user/makhshif/./assignRole.sh name open_mode
This runs fine
What is the problem with ssh command and how should I adjust these parameters so these can be accepted for the shell script assignRole.sh
< /oracle/oracle_user/makhshif/./assignRole.sh
This commands sends a content of that file to stdin. So obviously it can't process variables that you haven't send to remote machine. Just preprocess your script or create a script on remote machine and call it with arguments
Though it's even easier to pass variables like this:
ssh -o StrictHostKeyChecking=no -T $key "var1=$var1 var2=$var2" < /oracle/oracle_user/makhshif/./assignRole.sh name open_mode >> /oracle/oracle_user/dftest.txt
For example my function for executing update scripts on all cluster nodes:
# functions:
ssh_exec(){
local DESCR="$1"; shift
local SCRIPT="$1"; shift
local hosts=("$#")
echo =================================================
echo = $DESCR
echo = Going to execute $SCRIPT...
read -a res -p "Enter 'skip' to skip this step or press Enter to execute: "
if [[ $res = "skip" ]]
then
echo Skipping $SCRIPT...
else
echo Executing $SCRIPT...
for host in "${hosts[#]}"
do
local cur=${!host}
echo Executing $SCRIPT on $host - $cur...
sshpass -p "$rootpass" ssh -o "StrictHostKeyChecking no" root#${cur} \
"ns1=$ns1 ns2=$ns2 search=$search zoo1=$zoo1 zoo2=$zoo2 zoo3=$zoo3 node0=$node0 pass=$pass CURIP=$cur CURHOST=$host bash -s" \
<$SCRIPT >log-$SCRIPT-$cur.log 2>&1
echo Done.
done
echo =================================================
fi
}
Then I use it like this:
read -p "Please check that Solr started successfully and Press [Enter] key to continue..."
#Solr configset and collections:
ssh_exec "Solr configset and collections" script06.sh zoo1 zoo2 zoo3
This command executes script06.sh on 3 servers (zoo1,zoo2,zoo3)
As Sayan said, using < redirects the output of running the assignRole.sh script locally, but you want to execute that script on the remote host, with the arguments.
Pass the whole command as the final argument to ssh, in quotes:
ssh -o StrictHostKeyChecking=no -T $key "/oracle/oracle_user/makhshif/./assignRole.sh name open_mode" >> /oracle/oracle_user/dftest.txt
or split into multiple lines for readability:
ssh -o StrictHostKeyChecking=no -T $key \
"/oracle/oracle_user/makhshif/./assignRole.sh name open_mode" \
>> /oracle/oracle_user/dftest.txt

Automatic extract zipped files with passwords in the file name

I analyse data from being sent multiple ZIP files.
They are always in this format:
service_SC30COM_####_20191130_1834.zip
#### is a random number generated by the computer.
Password is SC30COM_####, which is always part of the file name.
Any suggestions on an automation to unzip in bulk?
There is no way to do it on the command prompt without any application.
You can check this
If your remove the password, the code you need, as explain in that microsoft article should be:
$shell=new-object -com shell.application
$CurrentLocation=get-location
$CurrentPath=$CurrentLocation.path
$Location=$shell.namespace($CurrentPath)
$ZipFiles = get-childitem *.zip
$ZipFiles.count | out-default
foreach ($ZipFile in $ZipFiles)
{
$ZipFile.fullname | out-default
$ZipFolder = $shell.namespace($ZipFile.fullname)
$Location.Copyhere($ZipFolder.items())
}
If you install any application that can run on command prompt, you can extract with password. As example, for a individual file, the maximum you will get on Windows 10 is:
PowerShell Expand-Archive -Path "C:\Users\Tuffy\Desktop\PowerShell
Expand-Archive -Path "C:\Users\Whatever\Desktop\service_SC30COM_####_20191130_1834.zip"
-DestinationPath "C:\Users\Whatever\Desktop"p" -DestinationPath "C:\Whatever\Tuffy\Desktop"
Hope it helps!
You can run the following code as a bash script:
#!/bin/bash
for FILE in *.zip
do
echo "Unzipping $FILE ..."
PASSWORD=$(echo $FILE | grep -o -P '(?<=service_)[A-Za-z0-9]*_[0-9]*(?=_)')
unzip -P $PASSWORD $FILE
done
Copy the code
Paste it into FILENAME.sh
Make it executable (chmod +x FILENAME.sh)
Put it besides the zip files
Run it (./FILENAME.sh)

GNU Parallel -q option causing BCP "unknown option" errors (different string quotes on local vs remote hosts)

Seeing very strange behavior where when when using gnu parallel to distribute export jobs using bcp from mssql-tools. It appears that when using the -q option for parallel, strings are interpreted differently on local host than on remote hosts.
Running only as a loop through files on local host, the bcp processes throws no errors
However, distributing the file exports with parallel, the bcp processes executing on the local host throw
/opt/mssql-tools/bin/bcp: unknown option
errors, while those executing on remote hosts (via a --sshloginfile param) finish successfully. The basic code being run looks like...
# setting some vars to pass
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
TO_SERVER_IP="-S 172.18.54.22"
DB="$dest_db" #TODO: enforce being more careful with this value
TABLE="$tablename" # MUST exist beforehand, case matters
USER=$(tail -n+1 $source_home/mssql-creds.txt | head -1)
PASSWORD=$(tail -n+2 $source_home/mssql-creds.txt | head -1)
DATAFILES="/some/path/to/files/"
TARGET_GLOB="*.tsv"
RECOMMEDED_IMPORT_MODE='-c' # makes a HUGE difference, see https://stackoverflow.com/a/16310219/8236733
DELIMITER="\\\t" # (currently not used) DO NOT use format like "'\t'", nested quotes seem to cause hard-to-catch error, want "\t" literal
....
bcpexport() {
filename=$1
TO_SERVER_ODBCDSN=$2
DB=$3
TABLE=$4 # MUST exist beforehand, case matters
USER=$5
PASSWORD=$6
RECOMMEDED_IMPORT_MODE=$7 # makes a HUGE difference, see https://stackoverflow.com/a/16310219/8236733
DELIMITER=$8 # not currently used
WORKDIR=$9
LOGDIR=${10}
....
/opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE
-t "\t" \
-e ${localfile}.bcperror.log
}
export -f bcpexport
parallelization_pernode=5
parallel -q -j $parallelization_pernode \
--sshloginfile $source_home/parallel-nodes.txt \
--env bcpexport \
bcpexport {} "$TO_SERVER_ODBCDSN" $DB $TABLE $USER $PASSWORD $RECOMMEDED_IMPORT_MODE $DELIMITER $workingdir $logdir \
::: $DATAFILES/$TARGET_GLOB #from hdfs nfs gateway
Looking at the bash interpretation of the processes (by running ps -aux | grep bcp on the hosts that parallelis given in the --sshloginfile) for the remote hosts we see...
/bin/bash -c bcpexport() { ... /opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" $TO_SERVER_ODBCDSN -U $USER -P $PASSWORD -d $DB $RECOMMEDED_IMPORT_MODE; -t "\t" -e ${localfile}.bcperror.log; ...
for the local host, the bash interpretation is...
/bin/bash -c bcpexport() { ... /opt/mssql-tools/bin/bcp "$TABLE" in "$localfile" $TO_SERVER_ODBCDSN -U $USER -P $PASSWORD -d $DB $RECOMMEDED_IMPORT_MODE; -t "\t" -e ${localfile}.bcperror.log; ...
that is, they look the same.
My current thought is that the "\t" in the bcp command is being interpreted in a problematic way. Debugging parallel without vs with the -q option we see...
$ parallel -j 5 --sshloginfile ./parallel-nodes.txt echo "Number {}: Running on \`hostname\`: \t" ::: 1 2 3 4 5
Number 4: Running on HW04.ucera.local: t
Number 1: Running on HW04.ucera.local: t
Number 2: Running on HW03.ucera.local: t
Number 5: Running on HW03.ucera.local: t
Number 3: Running on HW02.ucera.local: t
$ parallel -q -j 5 --sshloginfile ./parallel-nodes.txt echo "Number {}: Running on \`hostname\`: \t" ::: 1 2 3 4 5
Number 1: Running on `hostname`:
Number 4: Running on `hostname`:
Number 3: Running on `hostname`: \t
Number 2: Running on `hostname`: \t
Number 5: Running on `hostname`: \t
The bcp command needs the "\t" literal not the "t" literal (and I suspect several other similar string corruptions (also I do believe that \t is the default for bcp anyway, but this is just an example and want to keep \t for code clarity)), but not sure how to get this for both local and remote nodes or even why this behavior differs by remote vs local.
Basically, need the the strings to be exactly the same for both local and remote hosts even if strings have spaces or escape characters in them (note, I think this used to not be the case (have older script on other machines that don't have this problem))
Not sure if this is counts more as a parallel problem or a bcp problem (currently thinking something is going wrong with the -q option in parallel, but not sure). Anyone have any debugging suggestions or fixes? Ideas of what could be happening?
Firstly, the reason why hostname is not expanded is due to -q. It quotes the ` so that it does not expand.
Secondly, I think what you see is the different behaviours in built-in echo and /bin/echo. Built-in echo depends on the shell. Here I compare echo \\\\t in different shells:
$ parallel --onall --tag -S sh#lo,bash#lo,csh#lo,tcsh#lo,ksh#lo,zsh#lo echo \\\\t ::: a
bash#lo \t a
tcsh#lo a
sh#lo a
ksh#lo \t a
zsh#lo a
csh#lo \t a
That does not, however, get you closer to a solution. If I were you I would use env_parallel to copy the environment variables. And if the login shell on the remote systems are not the same as your shell, then set PARALLEL_SHELL to force using that shell.
So:
#!/bin/bash
env_parallel --session
# setting some vars to pass
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
:
:
PARALLEL_SHELL=bash env_parallel -q -j $parallelization_pernode ...
(no need to use neither --env nor 'export -f' when using 'env_parallel --session')
# Cleanup (not needed if this is the last line in the script)
env_parallel --end-session

how do I write SSH command output with only values I need

I am creating a VPS with the API provided for command line. The output of the command comes with several text inside which I don't need. This is my command.
The variables are predefined and work fine.
echo y | /usr/local/bin/CLICMD vm create --hostname=$VMNAME --domain=$srvdomain --cpu 1 --memory 1024 --image $image --datacenter=$dc --billing=hourly -n 100 > /dev/null 1>> /home/logs/createvps.log
When I run it, it gives me the following output in createvps.log file,
This action will incur charges on your account. Continue? [y/N]: id 11232312
created 2015-06-13T14:43:27-05:00
guid xxxxxx-r345-4323-8e3f-c8c04e18fad7
From the above output, I just need to have id (11232312) value stored in a mysql table. I know how to grab the value from log file and save in mysql.
My question is, how do I save just that id in the log file instead of all the other values/strings.
Thank you in advance.
Not sure what is exactly your question, but I guess this should help you:
echo y | /usr/local/bin/CLICMD vm create --hostname=$VMNAME \
--domain=$srvdomain --cpu 1 --memory 1024 --image $image \
--datacenter=$dc --billing=hourly -n 100 | \
grep -oE "id [0-9]+$" | grep -Eo "[0-9]+" >> /home/logs/createvps.log
Few notes to difference in your code and mine:
You do two redirection of stdout, one to /dev/null and one to your log, which is equivalent of doing just one redirection (writing in /dev/null is practically NOP).

Using expect to login to amazon server with .PEM file

I need to do the following:
Log into my amazon server
Change to a specific directory and run a script
The script executes an svn up, I need to be able to pass my username and password to this script.
I've read I might be able to do this with expect? Can I do the login via a shell script and then invoke expect to run the custom script?
Basically, I'm just looking for a good way to do this and would appreciate a pointer in the right direction.
You can use ssh to pass a shell commands to be run on remote Instance.
For example, here's how I check logs on multiple Servers:
#!/bin/bash
nas_servers=(
"ec2-xx-xx-xxx-xxx.ap-xxxx.compute.amazonaws.com"
"ec2-xx-xx-xxx-xxx.ap-xxxx.compute.amazonaws.com"
"ec2-xx-xx-xxx-xxx.ap-xxxx.compute.amazonaws.com"
"ec2-xx-xx-xxx-xxx.ap-xxxx.compute.amazonaws.com"
)
for s in "${nas_servers[#]}"
do
echo "Cheking $s:"
ret=$(ssh -i ~/pem/Key.pem "user#$s" bash << 'EOF'
files=/var/log/syslog*
for f in $files
do
if [[ ${f##*.} = 'gz' ]]; then
cmd=zcat
else
cmd=cat
fi
$cmd $f | egrep -wi 'error|warn|crit|fail'
done
EOF
)
if [[ -z $ret ]]; then
echo "No errors found."
else
echo "$ret"
fi
done