I am creating an SSIS package which needs to selectively loop through excel files in a folder that contain a specific keyword in their name (the keyword can appear anywhere in the name). This folder will include multiple files that do not include this keyword.
Bonus: I then need to use that same file name to create a date variable in SSIS (the file name will have the required components to create the date inside it). So far my efforts have failed, any help would be appreciated!
Create a Foreach File Loop Enumertor. In the FileSpec, use a wildcard, such as:
*SearchString*.csv or *SearchString*.xslx
You will probably want to select the Filename and Extension radio button. Map that to an SSIS package variable.
For the bonus, you can create another package variable with an expression based on the mapped variable and perform whatever string parsing you need in order to extract information from the file name.
Related
I'm trying to use oracle external tables to load flat files into a database but I'm having a bit of an issue with the location clause. The files we receive are appended with several pieces of information including the date so I was hoping to use wildcards in the location clause but it doesn't look like I'm able to.
I think I'm right in assuming I'm unable to use wildcards, does anyone have a suggestion on how I can accomplish this without writing large amounts of code per external table?
Current thoughts:
The only way I can think of doing it at the moment is to have a shell watcher script and parameter table. User can specify: input directory, file mask, external table etc. Then when a file is found in the directory, the shell script generates a list of files found with the file mask. For each file found issue a alter table command to change the location on the given external table to that file and launch the rest of the pl/sql associated with that file. This can be repeated for each file found with the file mask. I guess the benefit to this is I could also add the date to the end of the log and bad files after each run.
I'll post the solution I went with in the end which appears to be the only way.
I have a file watcher than looks for files in a given input dir with a certain file mask. The lookup table also includes the name of the external table. I then simply issue an alter table on the external table with the list of new file names.
For me this wasn't much of an issue as I'm already using shell for most of the file watching and file manipulation. Hopefully this saves someone searching for ages for a solution.
I would like to create this in SSIS package, wherein, it will loop in the list of files in a directory or folder and check each files if it exist in a SQL queried table. If the filename does not exist in the table, it will be archive in a different folder.
Thanks is advance for any help! =)
You could use a For each Loop container that can be pointed to the said directory or folder and then store the file name (Name and Extension, Fully Qualified or Name only depending on the requirement) in a variable eg. LoopVariable. Use a Execute SQL task inside the container that has a prepared statement like -
If exists (select * from [Files] where [FileName] = ?)Select 'Yes'
Else Select 'No' -
In the Parameter pass the LoopVariable and Result map to another variable FileExists. In the precedence Constraint editor use - expression - [User::FileExists] =="No" and connect to a file system task that moves the file to the archive folder.
Note: You might need to form the file path in case you are not getting the fully qualified name from the For loop container.
I've few excel source files in one folder in SSIS. I want to pull data from these excel files and load in to SQL tables.
My problem is I want to save all the files names one by one and want to create SQL table with exactly same name as filename
and then want to load each excel file in corresponding table.
Please help me how to create a package for this.
Jayvee has presented the high level view which is good enough! Let me add in bit detail.
I am assuming that you have dynamic Excel file connection.
Declare a variable and named it as FileName. And assign it the first file name which is available in the folder.
Place Foreach Loop Container and double click on it. Specify the Folder: and Files: as shown in image below.
In the same Foreach Loop Editor, go to Variable Mappings. Select Variable from drop down list. This is the same variable which we defined in first step. Set its Index to 0. Click OK.
Remaining task is same as Jayvee explained.
See this link for further help. And this for Result Set Property Not Set Correctly. I think setting ResultSet property to SingleRow will do the job.
your control flow should look like this:
I have a table with a column type of xml. I also have a directory that can have 0 to n number of xml documents. For each xml document, i need to insert a new row in the table and throw the xml into the xml column.
To fit with our clients needs, I need to perform this operation using an SSIS package. I plan to use a Stored Procedure to insert the xml, passing in the file path.
I've created the stored procedure and tested, it functions as expected.
My question is, how do I execute the stored procedure from an SSIS package for each xml document is a specific directory?
Thanks in advance for any help.
-
Basically you just need to loop through the files and get the full file paths to pass to the stored proc. This can be done easily using a For Each Loop and the ForEach File Enumerator. This page has a good description of how to set that up:
http://www.sqlis.com/post/Looping-over-files-with-the-Foreach-Loop.aspx
Within the loop then you just access the variable that is populated each time the loop executes (an XML file is found) and send it as a parameter into an Execute SQL Task (residing inside your For Eacu Loop container) to call your stored procedure. Here is an example of passing variables as parameters:
http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-ole-db-source-editor.aspx
You don't need to use a stored procedure for this. You can do all of this within an SSIS package. Here's how:
Have a For-Each Loop task read all available files in the folder. Put the full path of the file into a variable called XMLFileName
Inside the For-Each loop, use a Data-Flow task read the contents.
The OLE_SRC is reading from the same SQL Server and it's statement is SELECT GetDate() as CurrentDateTime
The DerivedColumn component creates a column called XMLFilePath with the full path of the XML file
The ImportColumn component is the one that does the magic. It will take the XMLFilePath as an input column, give it the LineageId of a new output column you create and it will import the full XML for you. Read more on how to set it up here:
http://www.bimonkey.com/2009/09/the-import-column-transformation/
Use the OleDB Destination to write to the table.
Basically, I can't find a way to combine two types. For example, supposing I want a connection to refer to a text file that is held in a path identified by one of the environment variables, and whose filename is a string form of the current day.
I can use SQL to set the filename, and an environment variable to set the path, but I can't seem to find a way to join the two into a full file path which can then be used as a Text File connection. Am I missing something?
Never mind, found a way to get the same effect here :
http://www.sqlservercentral.com/articles/DTS/2851/
with an ActiveX script task.