In SSIS 2005, I am using the FTP Task. I have a flow where when the package runs, it retrieves any files in a specific folder from FTP to a local folder.
Remote folder path is set by variable such as /root/abc/*abc.txt
The task works fine if there are files in that folder matching this criteria. If there are no files, the task fails with a file not found error!
How can I make SSIS not break the task in case this specific file not found error comes up simply becuase the remote folder had no matching files?
But, in case there is an error such as FTP server not able to login etc., then the task should throw the expected error.
Probably, you have found an answer to your question by now. Here is one possible way of achieving this. Script Task can be used to find the list of files present in an FTP folder path for a given pattern (say *.txt). Below example shows how this can be done.
Step-by-step process:
On the SSIS package, create an FTP Connection named FTP and also create 5 variables as shown in screenshot #1. Variable RemotePath contains the FTP folder path; LocalPath contains the folder where the files will be downloaed to; FilePattern contains the file pattern to find the list of files to download from FTP server; FileName will be populated by the Foreach loop container but to avoid FTP task design time error, it can be populated with / or the DelayValidation property on the FTP Task can be set to True.
On the SSIS package, place a Script Task, Foreach Loop container and FTP Task within the Foreach Loop container as shown in screenshots #2.
Replace the Main() method within the Script Task with the code under the Script Task Code section. Script Task will populate the variable ListOfFiles with the collection of files matching a given pattern. This example will first use the pattern *.txt, which yields no results and then later the pattern *.xls that will match few files on the FTP server.
Configure the Foreach Loop container as shown in screenshots #3 and #4. This task will loop through the variable **ListOfFiles*. If there are no files, the FTP task inside the loop container will not execute. If there are files, the FTP task inside the loop container will execute for the task for the number of files found on the FTP server.
Configure the FTP Task as shown in screenshots #5 and #6.
Screenshot #7 shows sample package execution when no matching files are found for the pattern *.txt.
Screenshot #8 shows the contents of the folder C:\temp\ before execution of the package.
Screenshot #9 shows sample package execution when matching files are found for the pattern *.xls.
Screenshot #10 shows the contents of the FTP remote path /Practice/Directory_New.
Screenshot #11 shows the contents of the folder C:\temp\ after execution of the package.
Screenshot #12 shows the package failure when provided with incorrect Remote path.
Screenshot #13 shows the error message related to the package failure.
Hope that helps.
Script Task Code:
C# code that can be used in SSIS 2008 and above.
Include the using statement using System.Text.RegularExpressions;
public void Main()
{
Variables varCollection = null;
ConnectionManager ftpManager = null;
FtpClientConnection ftpConnection = null;
string[] fileNames = null;
string[] folderNames = null;
System.Collections.ArrayList listOfFiles = null;
string remotePath = string.Empty;
string filePattern = string.Empty;
Regex regexp;
int counter;
Dts.VariableDispenser.LockForWrite("User::RemotePath");
Dts.VariableDispenser.LockForWrite("User::FilePattern");
Dts.VariableDispenser.LockForWrite("User::ListOfFiles");
Dts.VariableDispenser.GetVariables(ref varCollection);
try
{
remotePath = varCollection["User::RemotePath"].Value.ToString();
filePattern = varCollection["User::FilePattern"].Value.ToString();
ftpManager = Dts.Connections["FTP"];
ftpConnection = new FtpClientConnection(ftpManager.AcquireConnection(null));
ftpConnection.Connect();
ftpConnection.SetWorkingDirectory(remotePath);
ftpConnection.GetListing(out folderNames, out fileNames);
ftpConnection.Close();
listOfFiles = new System.Collections.ArrayList();
if (fileNames != null)
{
regexp = new Regex("^" + filePattern + "$");
for (counter = 0; counter <= fileNames.GetUpperBound(0); counter++)
{
if (regexp.IsMatch(fileNames[counter]))
{
listOfFiles.Add(remotePath + fileNames[counter]);
}
}
}
varCollection["User::ListOfFiles"].Value = listOfFiles;
}
catch (Exception ex)
{
Dts.Events.FireError(-1, string.Empty, ex.ToString(), string.Empty, 0);
Dts.TaskResult = (int) ScriptResults.Failure;
}
finally
{
varCollection.Unlock();
ftpConnection = null;
ftpManager = null;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
VB code that can be used in SSIS 2005 and above.
Include the Imports statement Imports System.Text.RegularExpressions
Public Sub Main()
Dim varCollection As Variables = Nothing
Dim ftpManager As ConnectionManager = Nothing
Dim ftpConnection As FtpClientConnection = Nothing
Dim fileNames() As String = Nothing
Dim folderNames() As String = Nothing
Dim listOfFiles As Collections.ArrayList
Dim remotePath As String = String.Empty
Dim filePattern As String = String.Empty
Dim regexp As Regex
Dim counter As Integer
Dts.VariableDispenser.LockForRead("User::RemotePath")
Dts.VariableDispenser.LockForRead("User::FilePattern")
Dts.VariableDispenser.LockForWrite("User::ListOfFiles")
Dts.VariableDispenser.GetVariables(varCollection)
Try
remotePath = varCollection("User::RemotePath").Value.ToString()
filePattern = varCollection("User::FilePattern").Value.ToString()
ftpManager = Dts.Connections("FTP")
ftpConnection = New FtpClientConnection(ftpManager.AcquireConnection(Nothing))
ftpConnection.Connect()
ftpConnection.SetWorkingDirectory(remotePath)
ftpConnection.GetListing(folderNames, fileNames)
ftpConnection.Close()
listOfFiles = New Collections.ArrayList()
If fileNames IsNot Nothing Then
regexp = New Regex("^" & filePattern & "$")
For counter = 0 To fileNames.GetUpperBound(0)
If regexp.IsMatch(fileNames(counter)) Then
listOfFiles.Add(remotePath & fileNames(counter))
End If
Next counter
End If
varCollection("User::ListOfFiles").Value = listOfFiles
Dts.TaskResult = ScriptResults.Success
Catch ex As Exception
Dts.Events.FireError(-1, String.Empty, ex.ToString(), String.Empty, 0)
Dts.TaskResult = ScriptResults.Failure
Finally
varCollection.Unlock()
ftpConnection = Nothing
ftpManager = Nothing
End Try
Dts.TaskResult = ScriptResults.Success
End Sub
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
Screenshot #5:
Screenshot #6:
Screenshot #7:
Screenshot #8:
Screenshot #9:
Screenshot #10:
Screenshot #11:
Screenshot #12:
Screenshot #13:
Related
Everything worked great from my Visual Studio on my PC running this from the Start button. When I build the executable and copied the executable to the production box and scheduled the job via SQL Server Agent on the production machine – everything worked fine to create the file, but the encryption bit does not work.
The gpg.exe is here on the production server: \sql2014\c$\Program Files (x86)\GnuPG\bin
The gpg is here on my PC: C:\Program Files (x86)\GnuPG\bin
The filename.csv gets created in the proper location ok - I tested with both these names
Dim Extract_File As String = “\sql2014\e$\Extracts\ProgramName\filename.csv”
‘Dim Extract_File As String = “E:\Extracts\ProgramName\filename.csv” ‘do to this from my PC I had to change the E: to a C:
This line calls the function:
FileEncrypted = Encrypt_File(Extract_File, Batch_Timestamp)
Private Function Encrypt_File(File_To_Encrypt As String, Batch_Timestamp As Date)
On Error GoTo Encrypt_File_Error
Dim Success As Boolean = False
Dim sourceName As String = File_To_Encrypt
Dim gpgProcess = New Process()
‘Test with working directory - no effect
‘gpgProcess.StartInfo.UseShellExecute = False
'gpgProcess.StartInfo.WorkingDirectory = "\\sql2014\c$\Program Files (x86)\GnuPG\bin\"
‘gpgProcess.StartInfo.FileName = "gpg.exe"
gpgProcess.StartInfo.FileName = \\sql2014\c$\Program Files (x86)\GnuPG\bin\gpg.exe ‘This works from my PC
‘gpgProcess.StartInfo.FileName = \\sql2014\c$\Program Files (x86)\GnuPG\bn\gpg.exe ‘If I change this path took the “i” out of bin I get an error: The system cannot find the file specified
gpgProcess.StartInfo.UseShellExecute = False
gpgProcess.StartInfo.CreateNoWindow = True
gpgProcess.StartInfo.Arguments = "--batch --yes --recipient reciptname --encrypt " & sourceName
gpgProcess.StartInfo.WindowStyle = ProcessWindowStyle.Hidden
gpgProcess.Start()
gpgProcess.WaitForExit()
If FileExists(sourceName & ".gpg") Then
Success = True
End If
Encrypt_File_Exit:
On Error Resume Next
‘gpgProcess.WaitForExit() moved this up to
gpgProcess.Close()
Return Success
Exit Function
Encrypt_File_Error:
Error_Handler("SomeModule.vb", "Encrypt_File", Err, System_Output, Batch_Timestamp)
Resume Encrypt_File_Exit
End Function
Any suggestions for how I can resolve this. When it worked on my PC it creates a filename.csv.gpg in the same directory as filename.csv. On the production server it does not create the gpg and it does not give a visible error message either.
This is how I solved this issue. I Installed the OpenPgpLib from the NuGet Package Manager and re-wrote this Function as shown here.
I created the .asc file from the Kleopatra tool and saved it in the location used in the pubkey in the code bit below. The OpenPgp is from the package.
Private Function Encrypt_File(File_To_Encrypt As String, Log_File As String, Batch_Timestamp As Date)
Dim Success As Boolean = False
Dim encryptthis As String = File_To_Encrypt
Dim thisencrypted As String = File_To_Encrypt & ".gpg"
Dim pubkey As String = "\\sql2014\c$\Data_Programs\MyDirectory\<thepublickeyfile>.asc"
Try
OpenPgp.EncryptFile(encryptthis, thisencrypted, pubkey, False, False)
If FileExists(thisencrypted) Then
Success = True
End If
Catch ex As Exception
App_Logger(Log_File, ex.StackTrace.ToString(), System_Output, Batch_Timestamp)
Success = False
End Try
Return Success
End Function
I have a XSD file which i am using to validate XML. The problem is i get an error. The error is not thrown when i run the code in local machine. But if i run the code in integration, the error is thrown.
Dim strSchemaPath As String = String.Empty
Dim xmlSettings As XmlReaderSettings = Nothing
Dim msStream As MemoryStream = Nothing
IsXMLValid = True
msStream = New MemoryStream(System.Text.ASCIIEncoding.UTF8.GetBytes(xmlRequest))
strSchemaPath = Path.Combine(System.AppDomain.CurrentDomain.BaseDirectory, "XSD\Input.xsd")
xmlSettings = New XmlReaderSettings()
xmlSettings.ValidationType = ValidationType.Schema
xmlSettings.Schemas.Add(Nothing, strSchemaPath)
There was no issues with access of the file. There was a issue with the files to be copied in the server path. We can manage the files in the properties. I just changed the file property to copy always and it worked.
I'm trying to create a VB.NET application which writes multiple lines of text into a text file, then starts an application and after the application started, deletes the text file.
How exactly can I realize that?
--
Edit:
I now got this code:
Dim iniFile As String = Application.StartupPath + "\settings.ini"
If System.IO.File.Exists(iniFile) = True Then
File.Delete(iniFile)
End If
If System.IO.File.Exists(iniFile) = False Then
File.Create(iniFile)
End If
Dim fileStr As String() = {"line1", "line2", "line3"}
File.WriteAllLines(iniFile, fileStr)
Dim p As Process = Process.Start(Application.StartupPath + "\app.exe")
p.WaitForInputIdle()
If System.IO.File.Exists(iniFile) = True Then
File.Delete(iniFile)
End If
The only problem I got, is that VS is telling me, the file is in use. Between creating and editing the file. Any ideas for that?
Your code is starting the app and then moving straight on to delete the ini file.
You need to wait for the process to exit first before you continue with deleting the ini file
E.g
{code to create ini file}
'Start the process.
Dim p As Process = Process.Start(Application.StartupPath + "\app.exe")
'Wait for the process window to complete loading.
p.WaitForInputIdle()
'Wait for the process to exit.
p.WaitForExit()
{code to delete ini file}
Full example here: https://support.microsoft.com/en-us/kb/305368
Just use File.WriteAllText
Note: As others already mentioned, you should check against True in your last If
If System.IO.File.Exists(Application.StartupPath + "\settings.ini") = False Then
File.Create(Application.StartupPath + "\settings.ini")
End If
Dim fileStr As String() = {"line1", "line2", "line3"}
System.IO.File.WriteAllText(Application.StartupPath + "\settings.ini", [String].Join(Environment.NewLine, fileStr))
Process.Start(Application.StartupPath + "\app.exe")
If System.IO.File.Exists(Application.StartupPath + "\settings.ini") = True Then
File.Delete(Application.StartupPath + "\settings.ini")
End If
Use File.WriteAllLines since it
Creates a new file, write the specified string array to the file, and then closes the file. [...] If the target file already exists, it is overwritten.
msdn
Also you should use Path.Combine to setup your path
Dim path as String = Path.Combine(Application.StartupPath, "settings.ini")
Dim fileStr As String() = {"line1", "line2", "line3"}
File.WriteAllLines(path, fileStr)
Use the Path.Combine for the Process.Start and File.Delete too.
For a long time I have been trying to debug why my parsing counts were off when downloading files to be parsed and been made to look really dumb about this. I did some debugging and found that the file I download when trying to decompress using GZipStream shows that it misses data from the original file. Here is my code for decompressing:
Using originalFileStream As FileStream = fileItem.OpenRead()
Dim currentFileName As String = fileItem.FullName
Dim newFileName = currentFileName.Remove(currentFileName.Length - fileItem.Extension.Length)
newFile = newFileName
Using decompressedFileStream As FileStream = File.Create(newFileName)
Using decompressionStream As GZipStream = New GZipStream(originalFileStream, CompressionMode.Decompress)
decompressionStream.CopyTo(decompressedFileStream)
Console.WriteLine("Decompressed: {0}", fileItem.Name)
decompressionStream.Close()
originalFileStream.Close()
End Using
End Using
End Using
Now what I do is return the newfile to the calling function and read the contents from there:
Dim responseData As String = inputFile.ReadToEnd
Now pasting the url in the browser and downloading from there and then opening using winrar I can see the data is not the same. Now this does not happen all the time as some files parse and decompress correctly. Each downloaded file has check counter to compare how many posts I am supposed to be parsing from it and that triggered me to see the mismatch in counts.
EDIT
Here is what I found in addition. If I read the problem file (as I said that only some files happen this way) by individual lines I will get all the data:
Dim rData As String = inputFile.ReadLine
If Not rData = "" Then
While Not inputFile.EndOfStream
rData = inputFile.ReadLine + vbNewLine + rData
End While
getIndividualPosts(rData)
End If
Now if I try to read an individual line from a file that is not problematic it will return nothing and so I will have to readtoEnd. Can anyone explain this odd behavior and is it related to the GZIPSTREAM or some error in my code.
I am trying to develop a package in SSIS 2005 and part of my process is to check if a file on the network is empty or not. If it is not empty, I need to pass a status of successful, otherwise, I need to pass a status of unsuccessful. I think I need a script task, but am not sure how to go about it. Any help is appreciated.
Create a connection to the flat file in the Connection Managers panel.
Under the Control flow tab, add a Data Flow Task.
Double click the Data flow task and add a Flat File Source and Row Count item.
In the Row Count properties, create a RowCount variable.
In the Control Flow tab, create control flow connections based on the result of the #RowCount.
There are two ways to do it:
If file empty means size = 0 you can create a Script Task to do the check:
http://msdn.microsoft.com/en-us/library/ms345166.aspx
If My.Computer.FileSystem.FileExists("c:\myfile.txt") Then
Dim myFileInfo As System.IO.FileInfo
myFileInfo = My.Computer.FileSystem.GetFileInfo("c:\myfile.txt")
If myFileInfo.Length = 0 Then
Dts.Variables["Status"].Value = 0
End If
End If
Otherwise, if file empty means no rows (flat file) you can use the a Row Count transformation after you reads the file. You can set a variable from the Row Count using the 'VariableName' property in Row Count editor and use it as a status.
Add a simple Script Task with the following code(C#) should do the trick:
String FilePath = (string)Dts.Variables["User::FilePath"].Value;
var length = new System.IO.FileInfo(FilePath).Length;
if (length == 0)
Dts.TaskResult = (int)ScriptResults.Success;
else
Dts.TaskResult = (int)ScriptResults.Failure;
This option will run a lot quicker than the accepted answer as it doesn't need to read the whole file, if you are cycling through a folder of files and some of them are large, in my case ~800mb, the accepted answer would take ages to run, this solution runs in seconds.
Yes, a Script task will do the job here. Add a using System.IO statement to the top of the script, then something along the lines of the following in the Main method will check the contents of the file.
public void Main()
{
String FilePath = Dts.Variables["User::FilePath"].Value.ToString();
String strContents;
StreamReader sReader;
sReader = File.OpenText(FilePath);
strContents = sReader.ReadToEnd();
sReader.Close();
if (strContents.Length==0)
MessageBox.Show("Empty file");
Dts.TaskResult = (int)ScriptResults.Success;
}
Edit: VB.Net version for 2005...
Public Sub Main()
Dim FilePath As String = Dts.Variables("User::FilePath").Value.ToString()
Dim strContents As String
Dim sReader As StreamReader
sReader = File.OpenText(FilePath)
strContents = sReader.ReadToEnd()
sReader.Close()
If strContents.Length = 0 Then
MessageBox.Show("Empty file")
End If
Dts.TaskResult = ScriptResults.Success
End Sub