Good with XML but beginner with Pentaho I am stumbling connecting the pieces.
Overall goal is to run an SQL query and format the output using XSLT into a nice email body. I use the XML Output block to get the SQL query result in XML, that is, it saves in XML format to a file, but it copies the data as CSV to the next block. I found this out by hooking up a temporary Text File Output block.
Why would a block named XML Output produce CSV output?
How can I get the XML to the XSL Translation block without needing to save to disk file in between?
Are there simpler ways of getting a nice email message body from an SQL SELECT query?
Here is the Pentaho Transformation where Report Totals does a simple SQL SELECT:
All steps in PDI output rows of data, with the same structure (field names, data types, etc.).
Each step is responsible to picking up those rows of data and doing something with them.
Your XML output step will pick up the rows of data coming from the table input step and write them as XML to an external file. But it looks like what you want is to create an XML field to use later in the XSL transformation step and then email. The step you're looking for is probably Add XML, not XML output.
Add XML will take the incoming rows and produce a set of XML rows as strings. Those can then be further manipulated by grouping them together into a single row, then inserted into a root XML element, then transformed and appended to the email body.
You can right click each of those steps and click on Preview to see what data is going out of each step, it should help you make sense of PDI's internals.
The Text file output outputs whatever data comes in as a CSV. As no XML fields are coming in, no XML is written out (the XML Output doesn't add a XML field to the data stream, only writes to an external file).
Hi I have looked for ways to do this but cant seem to get a clear answer. We have a requirement to Dump any changes to a table in SQL to a CSV File i.e Whenever anyone makes a change on the frontend to a customer record it must dump those changes to a CSV file for us to Update another System. Can anyone help me with an example either using SSIS or SQL Triggers.
I am new to SSIS, I am trying to load data from XML file into SQL server table. I have created the project and can transform and load data into table, but with one issue, below is sample of XML data
<EventLocationInfo>
<Facility>NY 31</Facility>
<Direction>eastbound</Direction>
<City>Cicero</City>
<County>Onondaga</County>
<State>NY</State>
<LocationDetails>
<LocationItem>
<Intersections>
<PrimaryLoc>I-81</PrimaryLoc>
<Article>area of</Article>
</Intersections>
</LocationItem>
<LocationItem>
<PointCoordinates Datum="NAD83">
<Lat>43.1755981445313</Lat>
<Lon>-76.1159973144531</Lon>
</PointCoordinates>
</LocationItem>
<LocationItem>
<AssociatedCities>
<PrimaryCity>Cicero</PrimaryCity>
<Article>area of</Article>
</AssociatedCities>
</LocationItem>
</LocationDetails>
</EventLocationInfo>
The result I am getting is like this
Is it possible to generate only one row instead of getting three different rows, if possible what Data Flow Transformation tool i can use to get this result.
Please help, thanks in advance.
Brijesh
Nothing technical here. Suppose I have a lot of different categorized data, and I would like to create a database out of it. Would someone literally hand plug in all that info with SQL code itself? Or do some people make a mock website just to input data? What are some of your strategies?
If there would be no way to do it automatically, then a mock website would be the way to go: you can even use it with more people at once, actually multiplying the input speed (as long as you don't mess up assigning each of them a different part of the data).
What format is your data in? And how much of it is there? If its Excel then SQL Server has tools to import it in. I'm not sure if MySQL has anything similar. Even if it doesn't one other technique I have used with Excel data is to use a formula to concatenate as required to generate the INSERT statements. Then just paste those into a query window and run that.
I wouldn't do a website for it unless I was building an admin site for it already and wanted to test that with the initial load.
Most databases have a way to do bulk inserts or have tools for data import.
My strategies normally involve such tools.
Here is an example of importing a CSV file to SQL Server.
Most database servers provide a way to import data from a variety of formats, you could look into that first.
If not, you could write a simple script or console application to parse your input data, and write out a SQL script to insert the data into appropriate tables.
For example, if you data was in a CSV file, you would parse each line in the file, and generate an insert statement to write out to a .sql file.
MyData.csv
1,2,3,'Test',4
2,3,4,'Test2,6
GeneratedInsert.sql
insert into table (col1,col2,col3,col4,col5) values (1,2,3,'Test',4)
insert into table (cal1,col2,col3,col4,col5) values (2,3,4,'Test2',6)
MySQL has a statement LOAD DATA INFILE that is intended for loading bulk data from flat files. It's easy to use and much faster than alternative methods.
But first you do have to use SQL to design tables with fields that match the field of your import data. That is, if you have some file with comma-separated data:
Titanic;1997;4 stars
Batman Begins;2005;5 stars
"Harry Potter and the Sorcerer's Stone";2001;3 stars
...
You would create a table:
CREATE TABLE Movies (
title VARCHAR(100) NOT NULL,
year YEAR NOT NULL
rating VARCHAR(10)
);
Then load data:
LOAD DATA INFILE 'movies.txt' INTO TABLE Movies
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"';
Most web languages have some sort of auto-scaffolding that you can quickly set up. Useful for admin work as well, if your site is hosted without direct access to DB.
Otherwise, yeah - write the SQL statements. Useful to bring a database up as part of your build process.
My current client stores all of their configuration information for the enterprise applications in a single table that holds XML. They then use a custom built front end to maintain the configuration values.
I'm writing a fairly straight-forward import process for them using SSIS. I need to make the connection strings and some other information configurable and they want me to use their table. It seems like SSIS expects a file though. Is there any way that I can point SSIS to retrieve its configuration values from an XML stream instead of a path to a file?
The configuration table that they use does not match the structure of a standard SSIS configuration table that you would get using SQL Server as a configuration source with the standard wizard.
Thanks for any advice!
You can retrieve values from the table, put it in variables, and using a script, transfer the varaibale values into the SSIS parameters.
Having the XML formatted just like the SSIS XML file is a huge bonus, though.
Is there a away to put a trigger on thier table to update the SQL Server config table or create a new XML document anytime an SSIS configuration is inserted or updated? Then you could use what you need and they could do what they need and all would be happy.