How to gather all file names in a folder and subdirectories into a SQL Server table - sql

Question

I would like to know how to do this.
For example:
I have c:/temp/.
Inside this temp folder I have various files and folders in various structure.
What would be the easiest way to gather all file names inside temp and its subdirectories and then insert them into a table?
I am planning the table structure will be simple.
It will have:
Primary key
Path and filename
CreatedDate
ModifiedDate
DeleteDate
So the table would look something like this:
Key | PathFilename | Modified | Created | Delete |
1 | c:\temp\fil7.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
2 | c:\temp\fi5e.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
3 | c:\temp\1ile.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
4 | c:\temp\2ile.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
5 | c:\temp\3ile.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
6 | c:\temp\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
7 | c:\temp\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
8 | c:\temp\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
9 | c:\temp\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
10 | c:\temp\folde1\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
11 | c:\temp\folde2\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
12 | c:\temp\folde4\file.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
13 | c:\temp\folder\fil5.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
14 | c:\temp\folder\fil6.txt | 2013/02/01 | 2013/02/01 | 1900/01/01|
Can I do this with a SSIS job? Or is there any other solution that can accomplish this task?
Is there any tutorial on how to do this step by step?
Thank you
ps: I have a filesystemWatcher vb.net program that will watch any created files and modified files
but for the initial start, I would like to fill up the table with files that already exists. I don't know if filesystemWatcher can do this initial task? can it?

Answer 1

I would create a Variable, FolderSource of type String and assign it your value of c:\temp.
While you can do all of this in a single Script Task which is an object on the Control Flow, I am going to describe how to do it with the Data Flow Task as that might be a better construct for learning how SSIS generally works. Drag a Data Flow Task onto the canvas. Double click on it.
Inside your Data Flow Task, add a Script Component. I add a reference to the Variable FolderSource as ReadOnly. In the Inputs and Outputs, I renamed the Output buffer to FS and added the columns below. Data types were 4 byte integer, string 255 and then date (DT_DATE).
public override void CreateNewOutputRows()
{
string src = Variables.FolderSource;
int key = 1;
System.IO.FileInfo fileInfo = null;
foreach (string currentFile in System.IO.Directory.EnumerateFiles(src, "*.*", SearchOption.AllDirectories))
{
fileInfo = new FileInfo(currentFile);
FSBuffer.AddRow();
FSBuffer.Key = key++;
FSBuffer.PathFilename = currentFile;
// Have UTC flavored methods too
FSBuffer.Created = fileInfo.CreationTime;
FSBuffer.Modified = fileInfo.LastWriteTime;
FSBuffer.Delete = new DateTime(1900, 1, 1);
}
}
That'll get the data streaming down your data flow. If you need to do anything with the data, you would add various components now.
Once you've manipulated the rows of data you'll need to land them somewhere. There are a host of destinations available but you'll probably only want to use the OLE DB Destination component. Connect the output of the Script Task, or any subsequent task(s) you used, to the destination. Double click on it and that will allow you to specify the database connection, the table name and the mapping of columns---in that order.
You probably don't have an OLE DB Connection Manager defined so click the Connection Manager button in the destination and create a new one. After creating a Connection Manager, you'll select the table where the data should reside. Then on the Columns tab, map the source columns (from the Script Component) to the destination (table).