Use findstr to find input from file a within file b, and send the output to a different file - scripting

I'm trying to write a batch script in Windows to take a list of IP addresses and ping them. Once a site doesn't respond, I want Windows to take all of the unresponsive IP addresses, and parse them through a comparison file that has the IP and physical street addresses of these systems. Once the unresponsive sites are parsed through the comparison file, I want the end result to be the matching info from the compare file for only the sites that are unresponsive. I already have script written for Linux that does this same thing, but I wanted a Windows version for some of the customers I work with who aren't Linux savvy.
Here is my script:
#Echo Off
Set "ServerList=C:\Users\<mylogin>\ip.txt"
Set "LogFile=C:\Users\<mylogin>\PingResults.txt"
If Not Exist "%ServerList%" Exit /B
>"%LogFile%" (For /F UseBackQ %%A In ("%ServerList%"
) Do Ping -n 1 %%A|Find "TTL=">Nul&&(Echo Yes [%%A] > Nul)||Echo No [%%A])
findstr /f:%LogFile sites.txt > Down.txt
The script itself seems to execute just fine, but it doesn't put anything in the final output file of Down.txt, I'm positive I have something wrong in my findstr command.
Below is my Linux script that does this exact same thing. Yeah it's clunky but it gets the job done:
#!/bin/bash
# Script to test ssh connectivity using expect script
rm -f results.txt
clear
echo "Please be patient while the script runs..."
while read user ip port pass; do
${PWD}/test_ssh_edge_device.sh $user $ip $port $pass >> results.txt
done < rekor_edge_device_list.txt
#This will boil down the results of the ping script so only the IP address is left
cat results.txt | grep -B1 time > refined.txt
cat refined.txt | grep -Eo "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" > refinedip.txt
#This will take the boiled down IP addresses and check them against the compare file, final output will be all system info for only downed systems
list=refinedip.txt
rm -f iplist.txt
exec 3<&0
exec 0<$list
while read line
do
cat compare.txt | grep $line >> iplist.txt
done
exec 0<&3
cat iplist.txt

Related

ssh to remote server with arguments to run scripts

I have lots of data that needs to be processed, and have access to 3 separate remote servers. The logic is to split up the data crunching among the 3 different servers instead of running on a single one. Note, that all three remote servers are able to point to a single directory, which is where I have the master scripts to process all of the data. The problem I am have is carrying over my arguments when I call different bash scripts.
For example, I have the master script which looks something like:
processing stuff
more stuff
# call the first script
$scriptdir/step1.csh $date $time $name
Within step1.csh, if I have something very simple where I am able to connect to one of the remote servers and output the hostname to a text file, such as:
#!/bin/bash
ssh name#hostname bash -c '
echo `hostname` > host.txt
I get the desired outcome, where 'host.txt' will be the hostname of the desired connected hostname. However, If step1.csh looks like:
#!/bin/bash
mydate=$1
mytime=$2
myname=$3
ssh name#hostname bash '
echo `hostname` > host.txt
echo ${mydate} > host.txt
I get the error saying that 'mydate: undefined variable'
Furthermore, If I do something along the lines of:
#!/bin/bash
mydate=$1
mytime=$2
myname=$3
ssh name#hostname "python /path/to/somewhere/to/run/${mydate}/and/${mytime}
It still runs on the local, and not remote server. What am I missing here?
So the first part:
#!/bin/bash
mydate=$1
mytime=$2
myname=$3
ssh name#hostname bash '
echo `hostname` > host.txt
echo ${mydate} > host.txt
The solution is:
#!/bin/bash
mydate=$1
mytime=$2
myname=$3
ssh -T name#hostname << EOF
echo `hostname` > host.txt
echo ${mydate} > host.txt
EOF
However, I am still having issues as in where I try to run a python script on the remote server; it is always ran on the local server.

Secure copying files from a remote server to local machine from a list in a text file

I have about a thousand files on a remote server (all in different directories). I would like to scp them to my local machine. I would not want to run scp command a thousand times in a row, so I have created a text file with a list of file locations on the remote server. It is a simple text file with a path on each line like below:
...
/iscsi/archive/aat/2005/20050801/A/RUN0010.FTS
/iscsi/archive/aat/2006/20060201/A/RUN0062.FTS
/iscsi/archive/aat/2013/20130923/B/RUN0010.FTS
/iscsi/archive/aat/2009/20090709/A/RUN1500.FTS
...
I have searched and found someone trying to do a similar but not the same thing here. The command I would like to edit is below:
cat /location/file.txt | xargs -i scp {} user#server:/location
In my case I need something like:
cat fileList.txt | xargs -i scp user#server:{} .
To download files from a remote server using the list in fileList.txt located in the same directory I run this command from.
When I run this I get an error: xargs: illegal option -- i
How can I get this command to work?
Thanks,
Aina.
You get this error xargs: illegal option -- i because -i was deprecated. Use -I {} instead (you could also use a different replace string but {} is fine).
If the list is remote, the files are remote, you can do this to retrieve it locally and use it with xargs -I {}:
ssh user#server cat fileList.txt | xargs -I {} scp user#server:{} .
But this creates N+1 connections, and more importantly this copies all remote files (scattered in different directories you said) to the same local directory. Probably not what you want.
So, in order to recreate a similar hierarchy locally, let's say everything under /iscsi/archive/aat, you can:
use cut -d/ to extract the part you want to be identical on both sides
use a subshell to create the command that creates the target directory and copies the file there
Thus:
ssh user#server cat fileList.txt \
| cut -d/ -f4- \
| xargs -I {} sh -c 'mkdir -p $(dirname {}); scp user#server:/iscsi/archive/{} ./{}'
Should work, but that's starting to look messy, and you still have N+1 connections, so now rsync looks like a better option. If you have passwordless ssh connection, this should work:
rsync -a --files-from=<(ssh user#server cat fileList.txt) user#server:/ .
The leading / is stripped by rsync and in the end you'll get everything under ./iscsi/archive/....
You can also copy the files locally first, and then:
rsync -a --files-from=localCopyOfFileList.txt user#server:/ .
You can also manipulate that file to remove for example 2 levels:
rsync -a --files-from=localCopyOfFileList2.txt user#server:/iscsi/archive .
etc.

how do I write SSH command output with only values I need

I am creating a VPS with the API provided for command line. The output of the command comes with several text inside which I don't need. This is my command.
The variables are predefined and work fine.
echo y | /usr/local/bin/CLICMD vm create --hostname=$VMNAME --domain=$srvdomain --cpu 1 --memory 1024 --image $image --datacenter=$dc --billing=hourly -n 100 > /dev/null 1>> /home/logs/createvps.log
When I run it, it gives me the following output in createvps.log file,
This action will incur charges on your account. Continue? [y/N]: id 11232312
created 2015-06-13T14:43:27-05:00
guid xxxxxx-r345-4323-8e3f-c8c04e18fad7
From the above output, I just need to have id (11232312) value stored in a mysql table. I know how to grab the value from log file and save in mysql.
My question is, how do I save just that id in the log file instead of all the other values/strings.
Thank you in advance.
Not sure what is exactly your question, but I guess this should help you:
echo y | /usr/local/bin/CLICMD vm create --hostname=$VMNAME \
--domain=$srvdomain --cpu 1 --memory 1024 --image $image \
--datacenter=$dc --billing=hourly -n 100 | \
grep -oE "id [0-9]+$" | grep -Eo "[0-9]+" >> /home/logs/createvps.log
Few notes to difference in your code and mine:
You do two redirection of stdout, one to /dev/null and one to your log, which is equivalent of doing just one redirection (writing in /dev/null is practically NOP).

Expect script SSH not successful

I have made two scripts:
This one fetches IP address & Hostnames:
#!/bin/bash
for i in `cat ~/script/hosts.txt`
do HOSTNAME=`echo $i|awk -F: '{print $1}'`
IP=`echo $i|awk -F: '{print $2}'`
TIMESTAMP=`date +%Y-%m-%d`
~/script/expect.sh $HOSTNAME $IP
done
This one does SSH into the devices:
#!/usr/bin/expect
set timeout 20
set HOSTNAME [lindex $argv 0]
set IP [lindex $argv 1]
exp_internal 1
spawn ssh -o StrictHostKeyChecking=no root#$IP
exit
I want to make a script to bakcup multiple device configurations.
Problem is that SSH is failing due to following errors:
$ ./main.sh
spawn ssh -o StrictHostKeyChecking=no root#10.102.82.235
: Name or service not knownname 10.102.82.235
spawn ssh -o StrictHostKeyChecking=no root#10.102.82.239
: Name or service not knownname 10.102.82.239
When I debug, I see the following error
spawn id exp4 sent <ssh: Could not resolve hostname 10.102.82.235\r\r: Name or
service not known\r\r\n>
: Name or service not knownname 10.102.82.235
I think the issue is due to these characters: "\r\r", "\r\r\n"
Is there any way I can filter these out?
Not an answer, but your shell script can use much improvement:
#!/bin/bash
while IFS=: read -r host ip; do
timestamp=$(date +%Y-%m-%d %T)
~/script/expect.sh "$host" "$ip"
done < ~/script/hosts.txt
Notes:
don't use for line in `cat file` to read the lines of a file -- a for loop reads words from a file
use $(...) for command substitution, not `...` -- improved readability, and easy to nest
don't use UPPERCASE_VARIABLES -- those should be reserved for the shell's use.
your (unused) TIMESTAMP variable actually contains a date, no time.
quote your "$variables" unless you can explain why you want them unquoted.

Redirect stderr to stdout in C shell

When I run the following command in csh, I got nothing, but it works in bash.
Is there any equivalent in csh which can redirect the standard error to standard out?
somecommand 2>&1
The csh shell has never been known for its extensive ability to manipulate file handles in the redirection process.
You can redirect both standard output and error to a file with:
xxx >& filename
but that's not quite what you were after, redirecting standard error to the current standard output.
However, if your underlying operating system exposes the standard output of a process in the file system (as Linux does with /dev/stdout), you can use that method as follows:
xxx >& /dev/stdout
This will force both standard output and standard error to go to the same place as the current standard output, effectively what you have with the bash redirection, 2>&1.
Just keep in mind this isn't a csh feature. If you run on an operating system that doesn't expose standard output as a file, you can't use this method.
However, there is another method. You can combine the two streams into one if you send it to a pipeline with |&, then all you need to do is find a pipeline component that writes its standard input to its standard output. In case you're unaware of such a thing, that's exactly what cat does if you don't give it any arguments. Hence, you can achieve your ends in this specific case with:
xxx |& cat
Of course, there's also nothing stopping you from running bash (assuming it's on the system somewhere) within a csh script to give you the added capabilities. Then you can use the rich redirections of that shell for the more complex cases where csh may struggle.
Let's explore this in more detail. First, create an executable echo_err that will write a string to stderr:
#include <stdio.h>
int main (int argc, char *argv[]) {
fprintf (stderr, "stderr (%s)\n", (argc > 1) ? argv[1] : "?");
return 0;
}
Then a control script test.csh which will show it in action:
#!/usr/bin/csh
ps -ef ; echo ; echo $$ ; echo
echo 'stdout (csh)'
./echo_err csh
bash -c "( echo 'stdout (bash)' ; ./echo_err bash ) 2>&1"
The echo of the PID and ps are simply so you can ensure it's csh running this script. When you run this script with:
./test.csh >test.out 2>test.err
(the initial redirection is set up by bash before csh starts running the script), and examine the out/err files, you see:
test.out:
UID PID PPID TTY STIME COMMAND
pax 5708 5364 cons0 11:31:14 /usr/bin/ps
pax 5364 7364 cons0 11:31:13 /usr/bin/tcsh
pax 7364 1 cons0 10:44:30 /usr/bin/bash
5364
stdout (csh)
stdout (bash)
stderr (bash)
test.err:
stderr (csh)
You can see there that the test.csh process is running in the C shell, and that calling bash from within there gives you the full bash power of redirection.
The 2>&1 in the bash command quite easily lets you redirect standard error to the current standard output (as desired) without prior knowledge of where standard output is currently going.
I object the above answer and provide my own. csh DOES have this capability and here is how it's done:
xxx |& some_exec # will pipe merged output to your some_exec
or
xxx |& cat > filename
or if you just want it to merge streams (to stdout) and not redirect to a file or some_exec:
xxx |& tee /dev/null
As paxdiablo said you can use >& to redirect both stdout and stderr. However if you want them separated you can use the following:
(command > stdoutfile) >& stderrfile
...as indicated the above will redirect stdout to stdoutfile and stderr to stderrfile.
xxx >& filename
Or do this to see everything on the screen and have it go to your file:
xxx | & tee ./logfile
What about just
xxx >& /dev/stdout
???
I think this is the correct answer for csh.
xxx >/dev/stderr
Note most csh are really tcsh in modern environments:
rmockler> ls -latr /usr/bin/csh
lrwxrwxrwx 1 root root 9 2011-05-03 13:40 /usr/bin/csh -> /bin/tcsh
using a backtick embedded statement to portray this as follows:
echo "`echo 'standard out1'` `echo 'error out1' >/dev/stderr` `echo 'standard out2'`" | tee -a /tmp/test.txt ; cat /tmp/test.txt
if this works for you please bump up to 1. The other suggestions don't work for my csh environment.