I found this site using a search which was a great help. I was able to modify part of a previous solution to do what I need but I still need some help. I need to do this in a batch file and was hoping to get a solution or two.
What I need to do is Transfer 1 to 10 zip files from my PC to multiple FTP Servers. I then need to check to make sure they are transferred successfully. On my PC I'm using regular MS FTP and the servers are using IndiFTPD. I have everything done except for the checking part. Some of the FTP Servers are Dial up and wireless and they get disconnected from time to time. So if I'm transferring say 3 files and the first one fails but the other two are successful my script lists it as successful.
I have was able to get a count of how many zip files are in the transfer folder and put it into a variable. The code for that is:
#echo off
for /f %%A in ('dir *.zip ^| find "File(s)"') do set cnt=%%A
echo Zip File count = %cnt%
What I'm having trouble with is getting the same information from the text log file of the FTP transfer and then comparing the two to make sure they are the same. Here is an example of a log of an FTP transfer I would make:
ftp> Connected to Host1.
open Host1 21
220 Connected to IndiFTPD
ftp> Username user
user1
331 Password required for Folder1.
230 User user1 logged in.
ftp> Hash mark printing On ftp: (2048 bytes/hash mark) .
ftp> hash
binary
200 Type is set to I.
ftp> mput *.zip
200 PORT command successful.
150 Opening BINARY mode data connection for 2012_07_01_03_40_1_3_TESTFTPs.zip.
###############################################################################################
226 [Bytes: 195046][Time: 1.13 s][Speed: 168.26 K/s]
ftp: 195046 bytes sent in Seconds Kbytes/sec.
0.78251.67200 PORT command successful.
150 Opening BINARY mode data connection for 2012_07_01_03_40_2_3_TESTFTPs.zip.
###############################################################################################
226 [Bytes: 195046][Time: 1.13 s][Speed: 168.41 K/s]
ftp: 195046 bytes sent in Seconds Kbytes/sec.
0.77253.31200 PORT command successful.
150 Opening BINARY mode data connection for 2012_07_01_03_40_3_3_TESTFTPs.zip.
###############################################################################################
226 [Bytes: 195046][Time: 1.14 s][Speed: 166.79 K/s]
ftp: 195046 bytes sent in Seconds Kbytes/sec.
ftp> 0.78249.42bye
221 Exit.
I would like to be able to use the %cnt% variable from the first part of the script to get the count of zip files I have. Then I would like to get able to count the number of "226 [Bytes" in the log file which is named terminal-ftp.txt. Then once I have both values in variable form I need to compare them to ensure I have 3 FTP files and 3 Successful 226 transfers. If they don't match then I would need to flag it up for a retry.
Get the 226 results:
for /f "usebackq tokens=3 delims=: " %%A in (`find /c "226 [Bytes" "terminal-ftp.txt"`) do set "xTTS=%%A"
echo Successful 226 Transfers = %xTTS%
Compare the results:
if "%xTTS%" EQU "%cnt%" echo The counts match
if "%xTTS%" NEQ "%cnt%" echo The counts do not match
For great Batch references:
command /?, Technet, Rob van de Woude, DosTips, and SS64
Related
I have a scheduled task with a .bat file that downloads some files from a web server every day by the morning then process the data and UPDATES a database. Then it triggers another .bat file to SELECT data and EXPORT to a .xls file.
The second .bat file is like this:
set a=%date:/=-%
del /q F:\file_path\file1_%a%.xls
del /q F:\file_path\file2_%a%.xls
echo %time%_%date%
cd /D D:\oracle\product\10.2.0\db_1\BIN
sqlplus usrname/psswd#ORCL #F:\select_path\select1.sql
timeout /t 30 /nobreak > nul
ren F:\file_path\file1.xls file1_%a%.xls
sqlplus usrname/psswd#ORCL #F:\select_path\select2.sql
timeout /t 30 /nobreak > nul
ren F:\file_path\file2.xls file2_%a%.xls
cd /D F:\KMB-SP\TI\Scripts\script_select
::Command to send file1 and file2 via e-mail.
But when I arrive at the office and check the progress, only the first .xls is done. So I have to run the second .bat manually and it runs perfectly.
What could be causing this?
Notes:
I put the timeout between the two SELECTs because, in the past, the code was stopping after the INSERT and didn't trigger the second .bat . My colleague said it could be execution exception. Puting a timeout would give time to end the INSERT properly.
Before, it used to make both SELECTs and then rename both files. Doing so, sometimes it worked, sometimes not, then I tried to change the order: select1, rename1, select2, rename2.
As we download files everyday, we concatenate the data on a single file called DT-date. The first code goes like this:
rem The data is downloaded and the files are organized in their files
if exist F:\path\DT-date (
Data_consolidation.exe
timeout /t 300 /nobreak > nul
F:\path\second_bat.bat
) else (exit)
As #William Robertson said, I tried echo exit right after the first SELECT, but again, it only extracted the first file and not the second one.
As #WilliamRobertson suggested, writing echo exit | before the sqlplus commands solved the problem.
Are there any specific problems with running Microsoft's BCP utility (on CentOS 7, https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-migrate-bcp?view=sql-server-2017) on multiple threads? Googling could not find much, but am looking at a problem that seems to be related to just that.
Copying a set of large TSV files from HDFS to a remote MSSQL Server with some code of the form
bcpexport() {
filename=$1
TO_SERVER_ODBCDSN=$2
DB=$3
TABLE=$4
USER=$5
PASSWORD=$6
RECOMMEDED_IMPORT_MODE=$7
DELIMITER=$8
echo -e "\nRemoving header from TSV file $filename"
echo -e "Current head:\n"
echo $(head -n 1 $filename)
echo "$(tail -n +2 $filename)" > $filename
echo "First line of file is now..."
echo $(head -n 1 $filename)
# temp. workaround safeguard for NFS latency
#sleep 5 #FIXME: appears to sometimes cause script to hang, workaround implemented below, throws error if timeout reached
timeout 30 sleep 5
echo -e "\nReplacing null literal values with empty chars"
NULL_WITH_TAB="null\t" # WARN: assumes the first field is prime-key so never null
TAB="\t"
sed -i -e "s/$NULL_WITH_TAB/$TAB/g" $filename
echo -e "Lines containing null (expect zero): $(grep -c "\tnull\t" $filename)"
# temp. workaround safeguard for NFS latency
#sleep 5 #FIXME: appears to sometimes cause script to hang, workaround implemented below
timeout 30 sleep 5
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
$TO_SERVER_ODBCDSN \
-U $USER -P $PASSWORD \
-d $DB \
$RECOMMEDED_IMPORT_MODE \
-t "\t" \
-e ${filename}.bcperror.log
}
export -f bcpexport
parallel -q -j 7 bcpexport {} "$TO_SERVER_ODBCDSN" $DB $TABLE $USER $PASSWORD $RECOMMEDED_IMPORT_MODE $DELIMITER \
::: $DATAFILES/$TARGET_GLOB
where $DATAFILES/$TARGET_GLOB constructs a glob that lists a set of files in a directory.
When running this code for a set of TSV files, finding that sometimes some (but not all) of the parallel BCP threads fail, ie. some files successfully copy to MSSQL Server
Starting copy...
5397376 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 154902 Average : (34843.8 rows per sec.)
while others output error message
Starting copy...
BCP copy in failed
Usually, see this pattern: a few successful BCP copy-in operations in the first few threads returned, then a bunch of failing threads return their output until run out of files (GNU Parallel returns output only when whole thread done to appear same as if sequential).
Notice in the code there is the -e option to produce an error file for each BCP copy-in operation (see https://learn.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017#e). When examining the files after observing these failing behaviors, all are blank, no error messages.
Only have seen this with the number of threads >= 10 (and only for certain sets of data (assuming has something to do with total number of files are files sizes, and yet...)), no errors seen so far when using ~7 threads, which further makes me suspect this has something to do with multi-threading.
Monitoring system resources (via free -mh) shows that generally ~13GB or RAM is always available.
May be helpful to note that the data bcp is trying to copy-in may be ~500000-1000000 records long with an upper limit of ~100 columns per record.
Does anyone have any idea what could be going on here? Note, am pretty new to using BCP as well as GNU Parallel and multi-threading.
No, no issues specific to the BCP program being run in multiple threads. You seem to be on the track of what I would say your issue is, system resources. Have you monitored system resources while increasing the number of threads? If anything, there is likely an issue with BCP executing properly when memory/cpu/network resources are low. Regarding the "-e" option, it is meant to output data errors. login errors, bad table names... many other errros are not reported in the file created with the -e option. When you get output using the "-e" option, you'll see info like "value truncated" and such... will give you line numbers and sample data that was at issue.
TLDR: Adding more threads to run concurrently to have bcp copy-in files of data seems to have the affect of overwhelming the endpoint MSSQL Server with write instructions, causing the bcp threads to fail (maybe timeing out?). When the number of threads becomes too many seems to depend on the size of the files getting copy-in'ed by bcp (ie. both the number of records in the file as well as the width of each record (ie. number of columns)).
Long version (more reasons for my theory):
1.
When running a larger number of bcp threads and looking at the processes started on the machine (https://clustershell.readthedocs.io/en/latest/tools/clush.html)
ps -aux | grep bcp
seeing a bunch of sleeping processes (notice the S, see https://askubuntu.com/a/360253/760862) as shown below (added newlines for readability)
me 135296 14.5 0.0 77596 6940 ? S 00:32 0:01
/opt/mssql-tools/bin/bcp TABLENAME in /path/to/tsv/1_16_0.tsv -D -S MyMSSQLServer -U myusername -P -d myDB -c -t \t -e /path/to/logfile
These threads appear to sleep for very long time. Further debugging into why these threads are sleeping suggests that they may in fact be doing their intended job (which would further imply that the problem may be coming from BCP itself (see https://stackoverflow.com/a/52748660/8236733)). From https://unix.stackexchange.com/a/47259/260742 and https://unix.stackexchange.com/a/36200/260742)
A process in S state is usually in a blocking system call, such as reading or writing to a file or the network, or waiting for another called program to finish.
(eg. writing to the MSSQL Server endpoint destination given to bcp in the ODBCDSN)
Your process will be in S state when it is doing reads and possibly writes that are blocking. Can also happen while waiting on semaphores or other synchronization primitives... This is all normal and expected, and not usually a problem... you don't want it to waste CPU while it's waiting for user input.
2. When running different sets of files of varying record-amount-per-file (eg. ranges of 500000 - 1000000 rows/file) and record-width-per-file (~10 - 100 columns/row), found that in cases with either very large data width or amounts, running a fixed set of bcp threads would fail.
Eg. for a set of ~33 TSVs with ~500000 rows each, each row being ~100 columns wide, a set of 30 threads would write the first few OK, but then all the rest would start returning failure messages. Incorporating a bit from #jamie's answer, the fact the the failure messages returned are "BCP copy in failed" errors does not necessarily mean it has do do with the content of the data in question. Having no actual content being written into the -e errorlog files from my process, #jamie's post says this
Regarding the "-e" option, it is meant to output data errors. login errors, bad table names... many other errros are not reported in the file created with the -e option. When you get output using the "-e" option, you'll see info like "value truncated" and such... will give you line numbers and sample data that was at issue.
Meanwhile, a set of ~33 TSVs with ~500000 rows each, each row being ~100 wide, and still using 30 bcp threads would complete quickly and without error (also would be faster when reducing the number of threads or file set). The only difference here being the overall size of the data being bcp copy-in'ed to the MSSQL Server.
All this while
free -mh
still showed that the machine running the threads still had ~15GB of free RAM remaining in each case (which is again why I suspect that the problem has to do with the remote MSSQL Server endpoint rather than with the code or local machine itself).
3. When running some of the tests from (2), found that manually killing the parallel process (via CTL+C) and then trying to remotely truncate the testing table being written to with /opt/mssql-tools/bin/sqlcmd -Q "truncate table mytable" on the local machine would take a very long time (as opposed to manually logging into the MSSQL Server and executing a truncate mytable in the DB). Again this makes me think that this has something to do with the MSSQL Server having too many connections and just being overwhelmed.
** Anyone with any MSSQL Mgmt Studio experience reading this (I have basically none), if you see anything here that makes you think that my theory is incorrect please let me know your thoughts.
I've search high and low looking for a way to display all text from FTP.exe to a richtextbox. so far i've only been able to do is display the output code. the idea is to run the test and display and capture to a file which hasn't been a problem except i can't seem to display all text as you would see it in Command.Hoping to see all text when done. Please Help!!
Here is he code:
Private Sub Rectangle1_Click(sender As Object, e As EventArgs) Handles Rectangle1.Click
Dim p As New Process()
With p
.StartInfo.Arguments = " -s:c:\dsl\ftptest\speed1.txt 65.40.220.20"
.StartInfo.CreateNoWindow = True
.StartInfo.FileName = "ftp"
.StartInfo.RedirectStandardError = True
.StartInfo.RedirectStandardOutput = True
.StartInfo.UseShellExecute = False
.Start()
Dim StErr As StreamReader = .StandardError
Dim StOut As StreamReader = .StandardOutput
While (Not StOut.EndOfStream)
Me.RichTextBox1.AppendText(String.Format("{0}", StOut.ReadLine() & vbCrLf))
End While
.WaitForExit()
End With
End Sub
End Class
Here is the output from the code:
User (65.40.220.20:(none)): Hash mark printing On ftp: (2048 bytes/hash mark) .
hash
get test.1meg
#
cd upload
put test.1meg
#
close
bye
Here is What I'm looking for:
C:\DSL\FTPTEST>call FTP -s:c:\dsl\FTPtest\speed1.txt 65.40.220.20
Connected to 65.40.220.20.
220-
This server is provided as a EMBARQ Speedtest server for DSL customers only.
Any other use is prohibited.
You may login using anonymous ftp and download the test files to determine your speed.
You may upload the same files to the upload directory to test your upload speed.
You may only upload the files that you previously downloaded from this server.
You cannot download anything from the upload directory.
Remember, some ftp programs measure speed in bytes per second.
DSL speeds are measured in bits per second. There are 8 bits in a byte.
If you can download at 64 kilobytes per second then that is the same as
512 kilobits per second.
220 65.40.220.20 FTP server ready
User (65.40.220.20:(none)):
331 Anonymous login ok, send your complete email address as your password.
230-
This server is provided as a EMBARQ Speedtest server for DSL customers only.
Any other use is prohibited.
You may login using anonymous ftp and download the test files to determine your speed.
You may upload the same files to the upload directory to test your upload speed.
You may only upload the files that you previously downloaded from this server.
You cannot download anything from the upload directory.
Remember, some ftp programs measure speed in bytes per second.
DSL speeds are measured in bits per second. There are 8 bits in a byte.
If you can download at 64 kilobytes per second then that is the same as
512 kilobits per second.
230 Anonymous access granted, restrictions apply.
ftp> hash
Hash mark printing On ftp: (2048 bytes/hash mark) .
ftp> get test.1meg
200 PORT command successful
150 Opening ASCII mode data connection for test.1meg (1048576 bytes)
#
#
#
ftp: 1048576 bytes received in 5.96Seconds 175.94Kbytes/sec.
ftp>
ftp> cd upload
250 CWD command successful
ftp> put test.1meg
200 PORT command successful
150 Opening ASCII mode data connection for test.1meg
#
#
#
226 Transfer complete.
ftp: 1048576 bytes sent in 5.98Seconds 175.23Kbytes/sec.
ftp>
ftp>
I think that you might be able to redirect the output of your command to a file. e.g, at the end of the command add (assuming that you have a directory c:\temp)
your command here > c:\temp\TestOutput.text
Then in your program, add a file system watcher to watch that file and load it into the textbox when it changes. If you're doing this lots of time then you might have to dynamically generate a filename and delete the files when no-longer needed.
I have a process that is kicked off by a scheduled batch file daily. I need to have error handling built in to restart the process if there is an error. All works great most days but I get a time out error once a month or so that is unavoidable. The process does not output an errorlevel to the bat file so I need to be able to parse the output file to determine if the process needs to restart.
I tried using the FOR /F function to pass the contents of line 12 as a variable to use in an IF statement but I have been unsuccessful. I can obviously skip to line 12 but then I am left dealing with the tokens of the remaining lines. Does anyone have any suggestions that I could try?
Output file when all is well:
(Line Numbers Added for Readability)
1 Pricing Script
2
3 ________________________________________________________________________
4
5 Retrieve Prices
6
7 Date of price file: 070912
8 Regular only
9 Connecting to server intdata.com
10 TCP/IP connection established
11
12 TySymb Interactive Data
+400 more lines
Output file when there is an error:
1 Pricing Script
2
3 ________________________________________________________________________
4
5 Retrieve Prices
6
7 Date of price file: 071012
8 Regular only
9 Connecting to server intdata.com
10 TCP/IP connection established
11 Time Out
12 General Time Out. The User ID and/or Password might be incorrect.
I would simply look for the error message in the output using FIND or FINDSTR. I wouldn't worry about the line number.
find "General Time Out. The User ID and/or Password might be incorrect." "yourFile.log" && (
echo Timeout occurred, you must put code here to restart
)
or
findstr /c:"General Time Out. The User ID and/or Password might be incorrect." "yourFile.log" && (
echo Timeout occurred, you must put code here to restart
)
This will test only line 12 of your file for a given string:
#echo off
for /f "skip=11 tokens=*" %%i in (your_log_filename) do (
echo %%i | find "General Time Out" >nul
goto check
)
:check
if errorlevel 1 (echo No Timeout occured!) else (echo Timeout occured)
In a DOS batch script, I'm running a single command on a remote (also windows) computer using plink. Formerly, this command was only run on the local machine, and was relying on the return code to determine success. Is there a way to easily get this information back through plink?
That's not possible with plink. The current consensus is to have the remote script echo its exit code to a log file, then use pscp to transfer the log file to the local machine.
See http://fixunix.com/ssh/74235-errorlevel-capturing-plink.html.
with plink 0.66
C:\Code>echo Y | "C:\Program Files (x86)\PuTTY\plink.exe" bob#myserver exit 42
C:\Code>echo %ERRORLEVEL%
42
Also for #John Wiersba's concern about when a connection cannot be made, this appears to be fixed
C:\CodeMisc>echo Y | "C:\Program Files (x86)\PuTTY\plink.exe" bob#garbageservername exit 42
Unable to open connection:
Host does not exist
C:\Code>echo %ERRORLEVEL%
1
Also note the piping of echo Y ... this enables you to accept the server fingerprint automatically (a little dangerous to say the least ... but our login server is load balanced, so you are always getting different fingerprints :( )
However as #LeonBloy notes, plink still has some connection conditions which return a zero exit code. If you know your exit code range and you don't have a good way of communicating back to windows via a file. You could either +3 to the exit code (if you know the exit code will never == 253-255) or you could apply a bitwise OR (I'd suggest exit $(($?|128)) - in bash).
Or if you don't care about the exact exit code, you could return 2 for success, and zero for failure. Thus a non-two exit code would indicate failure. In bash this would be: echo $((($?==0) << 1)). This would be by far the most robust general purpose solution, but you should make sure your exit code is logged for debug-ability.