Determine if file is empty (SSIS) - sql-server-2005

I am trying to develop a package in SSIS 2005 and part of my process is to check if a file on the network is empty or not. If it is not empty, I need to pass a status of successful, otherwise, I need to pass a status of unsuccessful. I think I need a script task, but am not sure how to go about it. Any help is appreciated.

Create a connection to the flat file in the Connection Managers panel.
Under the Control flow tab, add a Data Flow Task.
Double click the Data flow task and add a Flat File Source and Row Count item.
In the Row Count properties, create a RowCount variable.
In the Control Flow tab, create control flow connections based on the result of the #RowCount.

There are two ways to do it:
If file empty means size = 0 you can create a Script Task to do the check:
http://msdn.microsoft.com/en-us/library/ms345166.aspx
If My.Computer.FileSystem.FileExists("c:\myfile.txt") Then
Dim myFileInfo As System.IO.FileInfo
myFileInfo = My.Computer.FileSystem.GetFileInfo("c:\myfile.txt")
If myFileInfo.Length = 0 Then
Dts.Variables["Status"].Value = 0
End If
End If
Otherwise, if file empty means no rows (flat file) you can use the a Row Count transformation after you reads the file. You can set a variable from the Row Count using the 'VariableName' property in Row Count editor and use it as a status.

Add a simple Script Task with the following code(C#) should do the trick:
String FilePath = (string)Dts.Variables["User::FilePath"].Value;
var length = new System.IO.FileInfo(FilePath).Length;
if (length == 0)
Dts.TaskResult = (int)ScriptResults.Success;
else
Dts.TaskResult = (int)ScriptResults.Failure;
This option will run a lot quicker than the accepted answer as it doesn't need to read the whole file, if you are cycling through a folder of files and some of them are large, in my case ~800mb, the accepted answer would take ages to run, this solution runs in seconds.

Yes, a Script task will do the job here. Add a using System.IO statement to the top of the script, then something along the lines of the following in the Main method will check the contents of the file.
public void Main()
{
String FilePath = Dts.Variables["User::FilePath"].Value.ToString();
String strContents;
StreamReader sReader;
sReader = File.OpenText(FilePath);
strContents = sReader.ReadToEnd();
sReader.Close();
if (strContents.Length==0)
MessageBox.Show("Empty file");
Dts.TaskResult = (int)ScriptResults.Success;
}
Edit: VB.Net version for 2005...
Public Sub Main()
Dim FilePath As String = Dts.Variables("User::FilePath").Value.ToString()
Dim strContents As String
Dim sReader As StreamReader
sReader = File.OpenText(FilePath)
strContents = sReader.ReadToEnd()
sReader.Close()
If strContents.Length = 0 Then
MessageBox.Show("Empty file")
End If
Dts.TaskResult = ScriptResults.Success
End Sub

Related

Null Reference Exception when reading file vb.NET

So I am creating a program that needs to be able to read line by line from a .cfg (config) file, it can open it happily; here is the code:
OpenConfig.ShowDialog()
file = OpenConfig.FileName()
fileReader()
However when it tries to read the file, using this code:
Function fileReader()
Dim reader As New StreamReader(file)
Dim vLb As ListBox = shopTabs.SelectedTab.Controls.Item(10) 'Listbox Variable
For i = 0 To reader.Peek
textline(i) = reader.ReadLine()
vLb.Items.Add(i)
Next
Return True
End Function
It throws an exception at the line:
textline(i) = reader.ReadLine()
Any help would be greatly appreciated as I can't work out why it does so.
Your code could be simplified into the following code:
Using openConfig As New OpenFileDialog()
If openConfig.ShowDialog(Me) = DialogResult.OK Then
For Each s As String In File.ReadAllLines(openConfig.FileName)
ListBox1.Items.Add(s)
Next
End If
End Using
As I commented, your code does some things that are highly questionable and undoubtedly difficult to maintain, such as referencing a control by the index property.
I suspect that your project would benefit from using UserControls too, since I'm guessing you have the same controls placed in every tab (a ListBox is always control index #10?).

Stream Reader and Writer Conflict

I am making a class that is to help with saving some strings to a local text file (I want to append them to that file and not overwrite so that it is a log file). When I write with the streamwriter to find the end of the previous text, I get an error "the file is not available as it is being used by another process". I looked into this problem on MSDN and I got very little help. I tried to eliminate some variables so I removed the streamreader to check was that the problem and it was. When I tried to write to the file then it worked and I got no error so this made me come to the conclusion that the problem arose in the streamreader. But I could not figure out why?
Here is the code:
Public Sub SaveFile(ByVal Task As String, ByVal Difficulty As Integer, ByVal Time_Taken As String)
Dim SW As String = "C:/Program Files/Business Elements/Dashboard System Files/UserWorkEthic.txt"
Dim i As Integer
Dim aryText(3) As String
aryText(0) = Task
aryText(1) = Difficulty
aryText(2) = Time_Taken
Dim objWriter As System.IO.StreamWriter = New System.IO.StreamWriter(SW, True)
Dim reader As System.IO.StreamReader = New System.IO.StreamReader(SW, True)
reader.ReadToEnd()
reader.EndOfStream.ToString()
For i = 0 To 3
objWriter.WriteLine(aryText(reader.EndOfStream + i))
Next
reader.Close()
objWriter.Close()
End Sub
As Joel has commented on the previous answer it is possible to change the type of locking.
Otherwise building on what Neil has suggested, if to try to write to a file with a new reader it is difficult not to lose the information already within the file.
I would suggest you rename the original file to a temporary name, "UserWorkEthicTEMP.txt" for example. Create a new text file with the original name. Now; read a line, write a line, between the two files, before adding your new data onto the end. Finally Delete the temporary file and you will have the new file with the new details. If you have an error the temporary file will serve as a backup of the original. Some sample code below:
Change file names
Dim Line as string
line=Reader.readline
Do until Line=nothing
objwriter.writeline(line)
line=reader.readline
loop
add new values on the end and remove old file
You are trying to read and write to the same file and this is causing a lock contention. Either store the contents of the file into a variable and then write it back out including your new data to the file.
Psuedo
Reader.Open file
String content = Reader.ReadToEnd()
Reader.Close
Writer.Open file
Loop
Writer.Write newContent
Writer.Close

Using Variables in Script Task in SSIS

I am using a SSIS package with an Execute SQL Task connected to Script Task. The values from Execute SQL Task are stored in a variable and in the Script Task I use the following code to give XML file.
Public Sub Main()
If My.Computer.FileSystem.FileExists("C:\SAMPLE.xml") Then
Dts.TaskResult = ScriptResults.Failure
Else
My.Computer.FileSystem.WriteAllText("C:\SAMPLE.xml", Dts.Variables(0).Value.ToString, False)
Dts.TaskResult = ScriptResults.Success
''
End If
End Sub
I don't want to hardcode the path to XML. So I created two new variables FileName and FilePath with package as scope. How do I edit my VB.Net code to actually use these two variables.I tried this but did not work:
Sub Main()
If My.Computer.FileSystem.FileExists(Dts.Variables("FileDest").Value.ToString()) Else
My.Computer.FileSystem.WriteAllText(Dts.Variables("FileDest").Value.ToString(), Dts.Variables(0).Value.ToString, False)Dts.TaskResult = ScriptResults.Success
''
End If
End Sub
Ideally I want to use two variables one for name and one for path but when I tried with a single variable which combines both, it didn't work.
I suspect that part of your problem is using a numeric index into the Variables collection. Once you added the FileDest variable, there's no guarantee that whatever variable was being used to store the Execute SQL Task results is still the zeroeth one in the collection.
Assuming the variable with the Execute SQL Task results is named XmlQueryResults, the following should do the trick:
Public Sub Main()
Dim filePath As String = Dts.Variables("FileDest").Value
Dim xmlToWrite As String = Dts.Variables("XmlQueryResults").Value
If My.Computer.FileSystem.FileExists(filePath) Then
Dts.TaskResult = ScriptResults.Failure
Else
My.Computer.FileSystem.WriteAllText(filePath, xmlToWrite, False)
Dts.TaskResult = ScriptResults.Success
End If
End Sub
(Don't forget to add the FileDest variable to the ReadOnlyVariables in the Script Task Editor script tab.)

System.out of Memory Exception for String Builder in SSIS Script Task

I am using a VB script in SSIS Script Task to add header and Trailer to a flat file. The code was working fine until recently i came across a problem where the rows in the file are more than usual and resulting in a failure on script task with error`Error:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)
at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount)
at System.IO.StreamReader.ReadToEnd()
at System.IO.File.ReadAllText(String path, Encoding encoding)
at System.IO.File.ReadAllText(String path)`
Can any one help me in fixing the problem please.I think instead of "String Builder" i need to use other string related method. I am getting error at
fileContents.Append(File.ReadAllText(Dts.Connections("DestinationConnection").ConnectionString))
Here is my code:
Public Sub Main()
Dim fileContents As New StringBuilder()
Dim finalfile As String
Dim firstline As String
Dim lastline As String
Dts.VariableDispenser.LockForRead("FirstLine")
Dts.VariableDispenser.LockForRead("LastLine")
Dts.VariableDispenser.LockForRead("FileName")
firstline = CType(Dts.Variables("FirstLine").Value, String)
finalfile = CType(Dts.Variables("FileName").Value, String)
lastline= CType(Dts.Variables("LastLine").Value, String)
'Write header, then append file contents and write back out.
fileContents.AppendLine(String.Format("{0}", firstline))
fileContents.Append(File.ReadAllText(Dts.Connections("DestinationConnection").ConnectionString))
fileContents.AppendLine(String.Format("{0}", lastline))
File.WriteAllText(finalfile, fileContents.ToString())
Dts.TaskResult = ScriptResults.Success
End Sub
Well, one simple way would be to just avoid the StringBuilder: open a TextWriter with File.CreateText, write the first line, then write File.ReadAllText(...), then write the final line.
However, that will only save you some memory - it will roughly halve the memory required, as you won't need it in both the StringBuilder and a string (which is what I think will happen now).
A much better alternative would be to:
Open the writer
Write the header line
Open the other file for reading
Loop over the file, reading a chunk of characters at a time and writing it to the new file, until you're done
Close the other file implicitly (use a Using statement for this)
Write the trailing line
Close the write implicitly (use a Using statement)
That way even if you've got huge files, you only need a small chunk of data in memory at a time.
The problem is File.ReadAllText has limitations when it comes to reading a large file because the entire file is read into memory.
What you will need to do is replace the File.ReadAllText with reading the file line by line and append it accordingly.
EDITED FOR AN EXAMPLE:
Option Explicit
Dim oFSO, sFile, oFile, sText
Set oFSO = CreateObject("Scripting.FileSystemObject")
sFile = "your text file"
If oFSO.FileExists(sFile) Then
Set oFile = oFSO.OpenTextFile(sFile, 1)
Do While Not oFile.AtEndOfStream
sText = oFile.ReadLine
If Trim(sText) <> "" Then
fileContents.AppendLine(sText)
End If
Loop
oFile.Close
Else
WScript.Echo "The file was not there."
End If
It's possible you may still have an issue with the fileContents StringBuilder. The original error shown though was thrown from the File.ReadAllText method. So hopefully, this does the trick.
If not, I would just forget about the fileContents StringBuilder all together and write out the header. Then read from the file line by line and write it out line by line, then finally write the footer.
An alternative (and much more SSIS-like) solution would be to create a Data Flow Task that reads your existing file, pipes it through a Script Component that adds the header and footer, and writes it to the file system. Here's what it might look like in SSIS 2005:
The Script Component will be a Transformation with the SynchronousInputID of its output set to False, so that it can generate header and footer rows:
And the VB source of the transform should look something like this:
Public Class ScriptMain
Inherits UserComponent
Dim headerWritten As Boolean = False
Public Overrides Sub IncomingRows_ProcessInputRow(ByVal Row As IncomingRowsBuffer)
If Not headerWritten Then
WriteHeader()
End If
OutgoingRowsBuffer.AddRow()
OutgoingRowsBuffer.theLine = Row.theLine
End Sub
Public Overrides Sub FinishOutputs()
MyBase.FinishOutputs()
WriteFooter()
End Sub
Private Sub WriteHeader()
OutgoingRowsBuffer.AddRow()
OutgoingRowsBuffer.theLine = "The First Header Line"
headerWritten = True
End Sub
Private Sub WriteFooter()
OutgoingRowsBuffer.AddRow()
OutgoingRowsBuffer.theLine = "Here's a footer line"
OutgoingRowsBuffer.AddRow()
OutgoingRowsBuffer.theLine = "Here's another one"
End Sub
End Class
This lets you use the streaming capabilities of SSIS to your advantage.

Continue flow on certain specific Error in SSIS

In SSIS 2005, I am using the FTP Task. I have a flow where when the package runs, it retrieves any files in a specific folder from FTP to a local folder.
Remote folder path is set by variable such as /root/abc/*abc.txt
The task works fine if there are files in that folder matching this criteria. If there are no files, the task fails with a file not found error!
How can I make SSIS not break the task in case this specific file not found error comes up simply becuase the remote folder had no matching files?
But, in case there is an error such as FTP server not able to login etc., then the task should throw the expected error.
Probably, you have found an answer to your question by now. Here is one possible way of achieving this. Script Task can be used to find the list of files present in an FTP folder path for a given pattern (say *.txt). Below example shows how this can be done.
Step-by-step process:
On the SSIS package, create an FTP Connection named FTP and also create 5 variables as shown in screenshot #1. Variable RemotePath contains the FTP folder path; LocalPath contains the folder where the files will be downloaed to; FilePattern contains the file pattern to find the list of files to download from FTP server; FileName will be populated by the Foreach loop container but to avoid FTP task design time error, it can be populated with / or the DelayValidation property on the FTP Task can be set to True.
On the SSIS package, place a Script Task, Foreach Loop container and FTP Task within the Foreach Loop container as shown in screenshots #2.
Replace the Main() method within the Script Task with the code under the Script Task Code section. Script Task will populate the variable ListOfFiles with the collection of files matching a given pattern. This example will first use the pattern *.txt, which yields no results and then later the pattern *.xls that will match few files on the FTP server.
Configure the Foreach Loop container as shown in screenshots #3 and #4. This task will loop through the variable **ListOfFiles*. If there are no files, the FTP task inside the loop container will not execute. If there are files, the FTP task inside the loop container will execute for the task for the number of files found on the FTP server.
Configure the FTP Task as shown in screenshots #5 and #6.
Screenshot #7 shows sample package execution when no matching files are found for the pattern *.txt.
Screenshot #8 shows the contents of the folder C:\temp\ before execution of the package.
Screenshot #9 shows sample package execution when matching files are found for the pattern *.xls.
Screenshot #10 shows the contents of the FTP remote path /Practice/Directory_New.
Screenshot #11 shows the contents of the folder C:\temp\ after execution of the package.
Screenshot #12 shows the package failure when provided with incorrect Remote path.
Screenshot #13 shows the error message related to the package failure.
Hope that helps.
Script Task Code:
C# code that can be used in SSIS 2008 and above.
Include the using statement using System.Text.RegularExpressions;
public void Main()
{
Variables varCollection = null;
ConnectionManager ftpManager = null;
FtpClientConnection ftpConnection = null;
string[] fileNames = null;
string[] folderNames = null;
System.Collections.ArrayList listOfFiles = null;
string remotePath = string.Empty;
string filePattern = string.Empty;
Regex regexp;
int counter;
Dts.VariableDispenser.LockForWrite("User::RemotePath");
Dts.VariableDispenser.LockForWrite("User::FilePattern");
Dts.VariableDispenser.LockForWrite("User::ListOfFiles");
Dts.VariableDispenser.GetVariables(ref varCollection);
try
{
remotePath = varCollection["User::RemotePath"].Value.ToString();
filePattern = varCollection["User::FilePattern"].Value.ToString();
ftpManager = Dts.Connections["FTP"];
ftpConnection = new FtpClientConnection(ftpManager.AcquireConnection(null));
ftpConnection.Connect();
ftpConnection.SetWorkingDirectory(remotePath);
ftpConnection.GetListing(out folderNames, out fileNames);
ftpConnection.Close();
listOfFiles = new System.Collections.ArrayList();
if (fileNames != null)
{
regexp = new Regex("^" + filePattern + "$");
for (counter = 0; counter <= fileNames.GetUpperBound(0); counter++)
{
if (regexp.IsMatch(fileNames[counter]))
{
listOfFiles.Add(remotePath + fileNames[counter]);
}
}
}
varCollection["User::ListOfFiles"].Value = listOfFiles;
}
catch (Exception ex)
{
Dts.Events.FireError(-1, string.Empty, ex.ToString(), string.Empty, 0);
Dts.TaskResult = (int) ScriptResults.Failure;
}
finally
{
varCollection.Unlock();
ftpConnection = null;
ftpManager = null;
}
Dts.TaskResult = (int)ScriptResults.Success;
}
VB code that can be used in SSIS 2005 and above.
Include the Imports statement Imports System.Text.RegularExpressions
Public Sub Main()
Dim varCollection As Variables = Nothing
Dim ftpManager As ConnectionManager = Nothing
Dim ftpConnection As FtpClientConnection = Nothing
Dim fileNames() As String = Nothing
Dim folderNames() As String = Nothing
Dim listOfFiles As Collections.ArrayList
Dim remotePath As String = String.Empty
Dim filePattern As String = String.Empty
Dim regexp As Regex
Dim counter As Integer
Dts.VariableDispenser.LockForRead("User::RemotePath")
Dts.VariableDispenser.LockForRead("User::FilePattern")
Dts.VariableDispenser.LockForWrite("User::ListOfFiles")
Dts.VariableDispenser.GetVariables(varCollection)
Try
remotePath = varCollection("User::RemotePath").Value.ToString()
filePattern = varCollection("User::FilePattern").Value.ToString()
ftpManager = Dts.Connections("FTP")
ftpConnection = New FtpClientConnection(ftpManager.AcquireConnection(Nothing))
ftpConnection.Connect()
ftpConnection.SetWorkingDirectory(remotePath)
ftpConnection.GetListing(folderNames, fileNames)
ftpConnection.Close()
listOfFiles = New Collections.ArrayList()
If fileNames IsNot Nothing Then
regexp = New Regex("^" & filePattern & "$")
For counter = 0 To fileNames.GetUpperBound(0)
If regexp.IsMatch(fileNames(counter)) Then
listOfFiles.Add(remotePath & fileNames(counter))
End If
Next counter
End If
varCollection("User::ListOfFiles").Value = listOfFiles
Dts.TaskResult = ScriptResults.Success
Catch ex As Exception
Dts.Events.FireError(-1, String.Empty, ex.ToString(), String.Empty, 0)
Dts.TaskResult = ScriptResults.Failure
Finally
varCollection.Unlock()
ftpConnection = Nothing
ftpManager = Nothing
End Try
Dts.TaskResult = ScriptResults.Success
End Sub
Screenshot #1:
Screenshot #2:
Screenshot #3:
Screenshot #4:
Screenshot #5:
Screenshot #6:
Screenshot #7:
Screenshot #8:
Screenshot #9:
Screenshot #10:
Screenshot #11:
Screenshot #12:
Screenshot #13: