I have a bunch of different xml souces (over 100), and I am trying to pipe them all into an OLE DB via SSIS. I started out trying to use the Union All function, but since not all of the data that goes into each row of the database is of the same type, some of the values are entered in as "ignore" by SSIS, and come out as null. Each xml source has multiple outputs as well, so I am trying to avoid manually pointing each node of each xml source at a different db source, and then combining them. I know that you can change the input type of the data in the xml source under the advanced options, but there doesn't appear to be a way to default everything to one data type (string would work for me). Is there a way to use the union all feature to work for me, or a better way to do this? Thanks in advance.
Does your server have access to folder that contains xml files?
If it does, using SSIS for that is not nesessary. Just use OPENROWSET
Also you can make stored procedure that selects data from all xml sources and union all results. This SP you can use in your SSIS package.
Related
We want some of our customers to be able to export some data into a file and then we have a job that imports that into a blank copy of a database at our location. Note: a DBA would not be involved. This would be a function within our application.
We can ignore table schema differences - they will match. We have different tables to deal with.
So on the customer side the function would ran somethiug like:
insert into myspecialstoragetable select * from source_table
insert into myspecialstoragetable select * from source_table_2
insert into myspecialstoragetable select * from source_table_3
I then run a select * from myspecialstoragetable and get a .sql file they can then ship to me which we can then use some job/sql script to import into our copy of the db.
I'm thinking we can use XML somehow, but I'm a little lost.
Thanks
Have you looked at the bulk copy utility bcp? You can wrap it with your own program to make it easier for less sophisticated users.
Since it is a function within your application, in what language is the application front-end written ? If it is .NET, you can use Data Transformation Services in SQL Server to do a sample export. In the last step, you could save the steps into a VB/.NET module. If necessary, modify this file to change table names etc. Integrate this DTS module into your application. While doing the sample export, export it to a suitable format such as .CSV, .Excel etc, whichever format from which you will be able to import into a blank database.
Every time the user wants do an export, he will have to click on a button that would invoke the DTS module integrated into your application, that will dump the data to the desired format. He can mail such file to you.
If your application is not written in .NET, in whichever language it is written, it will have options to read data from SQL Server and dump them to a .CSV or text file with delimiters. If it is a primitive language, you may have to do it by concatenating the fields of every record, by looping through the records and writing to a file.
XML would be too far-fetched for this, though it's not impossible. At your end, you should have the ability to parse the XML file and import it into your location. Also, XML is not really suited if the no. of records are too large.
You probably think of a .sql file, as in MySql. In SQL Server, .sql files, that are generated by the 'Generate Scripts' function of SQL Server's interface, are used for table structures/DDL rather than the generation of the insert statements for each of the record's hard values.
I'm using a basic query to gather data from a few joined tables, and I need to be able to export the data to a CSV (or text) file in order to be imported into Excel. The query format is:
SELECT
Item1 as 'blah'
FROM
table1 JOIN table2
WHERE Condition
GROUP BY ...
HAVING ....
I have the proper output setup correctly through the query, so I'm only looking for a way to output it to a file. If it would be easier to use a stored procedure, then it would be no problem to throw that around the query. I'm just looking for something that can write the output to a file, WITHOUT using a third-party tool, as this needs to be moderately portable.
If you need more detail from the query, I can supply that (but it really is basic).
Use a BCP utility [MSDN]
That's what most commonly used.
If you have access to SSIS (through the Business Intelligence Development Studio), create a data flow task with an OLE DB Source going to a Flat File Destination. Or, you can go straight to Excel, if you don't want to worry about converting the delimited file.
What is the best practice when importing and validating an XML file to a single table (flattened) in SQL Server ?
I've a XML file which contains about 15 complex types which are all related to a single parent element.
The SSIS design could look like this:
But it's getting very complicated with all those (15) joins.
Is it maybe a better idea to just write T-SQL code to :
1) Import the XML into a column which is of the type XML and is linked to a XSD-schema.
2) Use this code:
TRUNCATE TABLE XML_Import
INSERT INTO XML_Import(ImportDateTime, XmlData)
SELECT GETDATE(), XmlData
FROM
(
SELECT *
FROM OPENROWSET (BULK 'c:\XML-Data.xml', SINGLE_BLOB) AS XMLDATA
) AS FileImport (XMLDATA)
delete from dbo.UserFlat
INSERT INTO dbo.UserFlat
SELECT
user.value('(UserIdentifier)', 'varchar(8)') as UserIdentifier,
user.value('(Emailaddress)', 'varchar(70)') as Emailaddress,
businessaddress.value('(Fax)', 'varchar(70)') as Fax,
employment.value('(EmploymentData)', 'varchar(8)') as EmploymentData,
-- More values here ...
FROM
XML_Import CROSS APPLY
XmlData.nodes('//user') AS User(user) CROSS APPLY
user.nodes('BusinessAddress') AS BusinessAddress(businessaddress) CROSS APPLY
user.nodes('Employment') AS Employment(employment)
-- More 'joins' here ...
to fill the 'UserFlat' table ?Some disadvantages are that you have to manually type the SQLcode, but the advantage here is that I have more direct control how the elements are processed and converted. But I don't know if there are any performance differences between processing XML in SSIS and processing the XML with T-SQL XML statements.
Note that some other requirements are:
Error handling : in case of an error, an email must be send to a person.
Able to process multiple input files with a specific file name pattern : XML_{date}_{time}.xml
Move the processed XML files to a different folder.
Please advice.
Based on the requirements that you have mentioned, I would say that you can use best of both the worlds (T-SQL & SSIS).
I feel that T-SQL gives more flexibility in loading the XML data that you have described in the question.
There are lot of different ways you can achieve this. Here is one possible option:
Create a Stored Procedure that would take the path of the XML file as input parameter.
Perform your XML data load operation using the T-SQL way which you feel is easier.
Use SSIS package to perform error handling, file processing, archiving and send email.
Use logging feature available in SSIS. It just requires simple configuration. Here is a samples that show how to configure logging in SSIS How to track status of rows successfully processed or failed in SSIS data flow task?
A sample mock up of your flow would be as shown below in the screenshot. Loop the files using Foreach Loop container. Pass the file path as parameter to Execute SQL Task, which in turn would call the T-SQL that you had mentioned. After processing the file, using the File System Task to move the file to an archive folder.
Sample used in SSIS reading multiple xml files from folder
shows how to loop through files using Foreach loop container. It loops through xml files but uses Data Flow Task because the xml files are in simpler format.
Sample used in How to send the records from a table in an e-mail body using SSIS package? shows how to send e-mail using Send Mail Task.
Sample used in How do I move files to an archive folder after the files have been processed? shows how to move files to an Archive folder.
Sample used in Branching after a file system task in SSIS without failing the package shows how to continue package execution even after a particular task fails. This will help you to proceed with package execution even if Foreach Loop fails so you can send email. Blue arrow in the screenshot indicates on completion of previous task.
Sample used in How do I pick the most recently created folder using Foreach loop container in SSIS package? shows how to perform pattern matching.
Hope that gives you an idea.
I have collection of object that contain several properties. I need to update this object list in the data base, now, i am looking for the best way to do it. Until now i use to work with xml, i created xml structure from the object list and send it to the data base and in the data base i parsed the xml and insert/update the tables i need.
Can you please suggest me better way to do that?
Sql 2005, MVC 2, C#4.
Thanks..
From what you've written, you'd probably be best to just loop through your collection and save each one using ADO.Net. That avoids the overhead of serializing your data to XML and then parsing it again on the database side.
That being said, unless this data is being generated by your software, you probably should be looking at something like SSIS or BCP for moving large amounts of data into SQL Server from whatever source your program is getting the data.
My current client stores all of their configuration information for the enterprise applications in a single table that holds XML. They then use a custom built front end to maintain the configuration values.
I'm writing a fairly straight-forward import process for them using SSIS. I need to make the connection strings and some other information configurable and they want me to use their table. It seems like SSIS expects a file though. Is there any way that I can point SSIS to retrieve its configuration values from an XML stream instead of a path to a file?
The configuration table that they use does not match the structure of a standard SSIS configuration table that you would get using SQL Server as a configuration source with the standard wizard.
Thanks for any advice!
You can retrieve values from the table, put it in variables, and using a script, transfer the varaibale values into the SSIS parameters.
Having the XML formatted just like the SSIS XML file is a huge bonus, though.
Is there a away to put a trigger on thier table to update the SQL Server config table or create a new XML document anytime an SSIS configuration is inserted or updated? Then you could use what you need and they could do what they need and all would be happy.