I have a table with a column type of xml. I also have a directory that can have 0 to n number of xml documents. For each xml document, i need to insert a new row in the table and throw the xml into the xml column.
To fit with our clients needs, I need to perform this operation using an SSIS package. I plan to use a Stored Procedure to insert the xml, passing in the file path.
I've created the stored procedure and tested, it functions as expected.
My question is, how do I execute the stored procedure from an SSIS package for each xml document is a specific directory?
Thanks in advance for any help.
-
Basically you just need to loop through the files and get the full file paths to pass to the stored proc. This can be done easily using a For Each Loop and the ForEach File Enumerator. This page has a good description of how to set that up:
http://www.sqlis.com/post/Looping-over-files-with-the-Foreach-Loop.aspx
Within the loop then you just access the variable that is populated each time the loop executes (an XML file is found) and send it as a parameter into an Execute SQL Task (residing inside your For Eacu Loop container) to call your stored procedure. Here is an example of passing variables as parameters:
http://geekswithblogs.net/stun/archive/2009/03/05/mapping-stored-procedure-parameters-in-ssis-ole-db-source-editor.aspx
You don't need to use a stored procedure for this. You can do all of this within an SSIS package. Here's how:
Have a For-Each Loop task read all available files in the folder. Put the full path of the file into a variable called XMLFileName
Inside the For-Each loop, use a Data-Flow task read the contents.
The OLE_SRC is reading from the same SQL Server and it's statement is SELECT GetDate() as CurrentDateTime
The DerivedColumn component creates a column called XMLFilePath with the full path of the XML file
The ImportColumn component is the one that does the magic. It will take the XMLFilePath as an input column, give it the LineageId of a new output column you create and it will import the full XML for you. Read more on how to set it up here:
http://www.bimonkey.com/2009/09/the-import-column-transformation/
Use the OleDB Destination to write to the table.
Related
I've few procedures(sql server) which produce output and I manually save them in text files with headers.
I need a generic SSIS package which takes procedure_name, output_filename as inputs and produce ouput text file.
As different procedure produces different columns, I need to handle metadata dynamically.
Ex: proc1 produces following result
col1|col2
a|b
proc2 produces following result:
col1|col2|col3
1|2|a
One solution was to change all the procedures to produce output as a single row, thus SSIS treats it as one column and produces output file.
Ex:
'col1|col2|cole' AS rowdata
'1|2|a' AS rowdata
However, there are quite a few procedures and changing every procedure will take lot of time and might end-up in human errors.
So, I'd like to know, if there is a way in SSIS, where I could handle metadata dynamically and produce output file.
You can look into BiML, which uses meta data to dynamically build and then execute packages.
Well, it is hardly possible in traditional SSIS package, where all outputs are defined at design time and validated at runtime. Altering its number leads to validation errors and stops package execution.
You can play a trick with SQL Execute Task and Script Component. Exec SQL runs SP and returns SP result as 'Full Result Set' stored in an Object variable. Then at Script task you open this variable which is in fact an ADO recordset; it can be read by the System.Data.OleDb.OleDbDataAdapter. For example:
// Set up the DataAdapter to extract the data,
// and the DataTable object to capture those results
OleDbDataAdapter da = OleDbDataAdapter();
DataTable dt = DataTable();
// Extract the data from the object variable into the table
da.Fill(dt, Variables.vResults);
Then you can work with DataSet and flatten into single variable for later use. Or - create and save file in C# script directly.
I have an SSIS package that takes in (through a package parameter) a value, passes it into a script task via a script variable (readonly variable), converts it to another value inside the script task, and finally writes that value out to another script variable (readwrite variable). There are no other SSIS modules in the package aside from the one script task.
What I would like to do, from outside the package (via SQL) is:
Call the SSIS package, passing in a value for my parameter and variable
Get the value of the read/write variable that is determined at the end of the script task execution
I've got step #1 working, just can't figure out #2.
How do I get the value of package variable in an SSIS package after it has executed? Is it accessible? Is it stored anywhere or can I store it somewhere in the SSIS catalog? I've tried to see if it's stored in the SSISDB.[catalog].executions table somewhere, but it doesn't seem to be.
Do I need to write that script variable to a package parameter in order to see it from SQL after execution? Could I then perhaps see it by using EXEC [SSISDB].[catalog].get_parameter_values, or does that only show parameter values before package execution? Am I going about this completely the wrong way?
Thanks in advance!
What I would do is add one last step to the package to write the value of the variable to a table.
Then you can retrieve the value from the table via SQL.
You can either truncate the table every time, or keep a permanent history associated with each time the package runs.
I have a job that I want to run that passes a variable to an ssis package. The variable is a filename but the filename changes daily. I have an access front end that the user enters the filename into. The access program runs a stored procedure which writes the filename to a temp table and then runs the job. I would like the job to query that table for the filename and pass it along to my package variable.
I can get the job to work using a static filename. On the set values tab I used the property path \Package.Variables[User::FileName] and the value \myserver......\filename.txt. But I don't know how to replace that filename with the results of the query
Thanks in advance.
Scott
I may have spoke too soon. The data source saved in my job step was still an value in my package. I had removed the value but didn't re-import the package to SQL. Now that I did that it is not importing anything at all.
I ended up creating an Execute SQL Task in my package that assigns the value in the temp table to my package variable.
I'm storing large (varchar(max)) SQL scripts in a table. I'd like to execute the scripts in an SSIS package.
Looking at other posts on this site it's easy enough to get the varchar(max) into an object variable. But then what to do? Is there a way for an Execute Sql Task (SQLSourceType of Variable) to specify an Object variable rather than a String variable?
Is there an approach that will work?
Here's how I might approach it:
Add a Data Flow task to your control flow
Add a Source (ADO.NET) that connects to your database
Create a Package level Object variable (for the next step)
Add a Recordset destination that populates your data into the Object variable created in the previous step
Back on the control flow:
Create a package level String variable for the "current" query (see next step)
Add a For Each ADO.NET enumerator
Connect the previous Data Flow task to the For Each task
Configure the For Each to use the Object variable as a source, and to store the column index with the SQL into the String variable
Add an Execute SQL task inside the For Each task
Configure it to execute a SQL Command from Variable, and pick the string variable containing the current query
Basically it will collect the queries from the table, then for each collected query, assign it to a variable, and then the Execute SQL command can pull the command text from that variable.
What is the best practice when importing and validating an XML file to a single table (flattened) in SQL Server ?
I've a XML file which contains about 15 complex types which are all related to a single parent element.
The SSIS design could look like this:
But it's getting very complicated with all those (15) joins.
Is it maybe a better idea to just write T-SQL code to :
1) Import the XML into a column which is of the type XML and is linked to a XSD-schema.
2) Use this code:
TRUNCATE TABLE XML_Import
INSERT INTO XML_Import(ImportDateTime, XmlData)
SELECT GETDATE(), XmlData
FROM
(
SELECT *
FROM OPENROWSET (BULK 'c:\XML-Data.xml', SINGLE_BLOB) AS XMLDATA
) AS FileImport (XMLDATA)
delete from dbo.UserFlat
INSERT INTO dbo.UserFlat
SELECT
user.value('(UserIdentifier)', 'varchar(8)') as UserIdentifier,
user.value('(Emailaddress)', 'varchar(70)') as Emailaddress,
businessaddress.value('(Fax)', 'varchar(70)') as Fax,
employment.value('(EmploymentData)', 'varchar(8)') as EmploymentData,
-- More values here ...
FROM
XML_Import CROSS APPLY
XmlData.nodes('//user') AS User(user) CROSS APPLY
user.nodes('BusinessAddress') AS BusinessAddress(businessaddress) CROSS APPLY
user.nodes('Employment') AS Employment(employment)
-- More 'joins' here ...
to fill the 'UserFlat' table ?Some disadvantages are that you have to manually type the SQLcode, but the advantage here is that I have more direct control how the elements are processed and converted. But I don't know if there are any performance differences between processing XML in SSIS and processing the XML with T-SQL XML statements.
Note that some other requirements are:
Error handling : in case of an error, an email must be send to a person.
Able to process multiple input files with a specific file name pattern : XML_{date}_{time}.xml
Move the processed XML files to a different folder.
Please advice.
Based on the requirements that you have mentioned, I would say that you can use best of both the worlds (T-SQL & SSIS).
I feel that T-SQL gives more flexibility in loading the XML data that you have described in the question.
There are lot of different ways you can achieve this. Here is one possible option:
Create a Stored Procedure that would take the path of the XML file as input parameter.
Perform your XML data load operation using the T-SQL way which you feel is easier.
Use SSIS package to perform error handling, file processing, archiving and send email.
Use logging feature available in SSIS. It just requires simple configuration. Here is a samples that show how to configure logging in SSIS How to track status of rows successfully processed or failed in SSIS data flow task?
A sample mock up of your flow would be as shown below in the screenshot. Loop the files using Foreach Loop container. Pass the file path as parameter to Execute SQL Task, which in turn would call the T-SQL that you had mentioned. After processing the file, using the File System Task to move the file to an archive folder.
Sample used in SSIS reading multiple xml files from folder
shows how to loop through files using Foreach loop container. It loops through xml files but uses Data Flow Task because the xml files are in simpler format.
Sample used in How to send the records from a table in an e-mail body using SSIS package? shows how to send e-mail using Send Mail Task.
Sample used in How do I move files to an archive folder after the files have been processed? shows how to move files to an Archive folder.
Sample used in Branching after a file system task in SSIS without failing the package shows how to continue package execution even after a particular task fails. This will help you to proceed with package execution even if Foreach Loop fails so you can send email. Blue arrow in the screenshot indicates on completion of previous task.
Sample used in How do I pick the most recently created folder using Foreach loop container in SSIS package? shows how to perform pattern matching.
Hope that gives you an idea.