I have a program (flex) that queries a database and gets back a dataset (value, timestamp). In the program I then put each value in the set through an algorithm to get a new value resulting in all the values being transformed. Instead of having to do this transformation of data in my program I would like mysql to do it and send the results back. Essentially, I would like to do a SELECT statement that returns a modified dataset.
Assuming the computation can only be done in a stored procedure or function, yes - as long as you are using MySQL 5.0+. I recommend reading this article on MySQL stored procedures & functions.
Related
I have a SQL Script with multiple drop & create DDL(Create tables As Select *), I want to run them at one go. I am quite new to informatica powercenter, can some one provide the process of using SQL transformation for BigQuery in informatica.
Sample Query:-
drop table if exists sellout.account_table;
CREATE TABLE sellout.account_table
AS
SELECT * FROM
sellout.account_src
WHERE
UPPER(account_name) IN ('RANDOM');
Similar to the above queries i have around 24 SQL's in a script.
I want to run them at once and later make them as part of informatica job.
If the "PowerExchange Google BigQuery" server and client are installed and after executing the infasetup.bat(sh) validateandregisterallfeatures, the mappings would be opened/exported successfully.
Here are some FAQs that might be handy for you:
Q: Why are the output fields in SQL Transformation not seen?
A: Stored Procedure selected in the SQL Transformation must have output parameters declared. Else it would not have output fields other than default Return Code column.
Q: A set of columns are displayed as result while running the Stored Procedure, however, you still do not see the same columns as output in SQL Transformation. Why?
A: Columns seen in the output might not be defined/declared as output parameters in the Stored Procedure. Procedure might have 'SELECT * FROM' like statement, which retrieves the data when the procedure is run from DB UI and a similar result could be seen when the procedure is run programmatically.
However, to call the same procedure from SQL Transformation, explicitly declared output parameters should be present as the transformation imports the metadata of the proc when selected. Unless you declare the output parameters explicitly in the procedure, it cannot be seen as output in the transformation.
Q: Is it necessary to have input/output parameters in Stored Procedure to call it from SQL Transformation?
A: Yes, it is necessary to have input/output parameters in Stored Procedure if it is not having default ones. As these parameters appear as input/output fields in SQL transformation, without these Mapping becomes invalid.
Q: I have SELECT statement in the procedure, does the SQL transformation can push this to next transforamtion?
A: Approprioate output parameters are required for this to work.
I have a table with a 'query' field containing a select sql and another 'parameters' field containing the sql parameters. I have already merged these two fields into a new field containing a correct select sql statement. Now I need to execute this new field containing select sql, get the return from select (the output fields) and generate an excel file.
In your transformation, you can use the Execute row SQL script (under scripting), which does exactly what you want, executing the SQL contained in the selected input field for each row passed to the step.
Please do consider if that's really what you want. It works fine for a small set or complicated logic, but is very inefficient compared to batch inserts and updates.
If your SQL statements do look like "INSERT [some data]" and you have many rows to process, consider splitting the streams with filter steps and applying calculations or constants to each case to set the values correctly, then directing all to a table output step.
I am using pentaho for data migration testing. I have set a "table input" step where many parts of the query inside "table inputs" are variables. I have been looking for a way to capture that query after it gets executed during runtime.
I was wondering if there is any specific system log variables for sql or is it to do with metadata. need help! Thanks
Maybe the following approach will help:
We assume a transformation reading a CSV file to get the dynamic portion of the SELECT statement (e.g. the columns) and setting the variable columns with it.
The second transformation uses this variable to generate the SELECT statement and store it into the variable sql_statement.
In the main transformation we use ${sql_statement} as the SELECT statement of the table input and write the data to an output file (that's the business process so to say). From the same input we copy the output to another path. There we add the current time as a field (use element "Get system data") and we add the generated SQL statement, join them as a cartesian product and group the result by the sql_statement. That way we can compute the first time and the last time that the statement was used. These results are written to a text file.
The last thing we need is a job calling the three transformations sequentially.
This is a sample output:
sql_statement;min_time;max_time
SELECT my_column FROM test_table;2014/05/08 00:41:21.143;2014/05/08 00:41:21.144
Thank you Marcus! I did some thing similar.
It works. awesome.
I gathered parts of queries from table field where they were kept and formed a full query in javascript. After that full query will be sent as parameter to a transformation that will run and log the query.
My project is in Visual Foxpro and I use MS SQL server 2008. When I fire sql queries in batch, some of the queries don't execute. However, no error is thrown. I haven't used BEGIN TRAN and ROLLBACK yet. What should be done ??
that all depends... You don't have any sample of your queries posted to give us an indication of possible failure. However, one thing I've had good response with from VFP to SQL is to build into a string (I prefer using TEXT/ENDTEXT for readabilty), then send that entire value to SQL. If there are any "parameter" based values that are from VFP locally, you can use "?" to indicate it will come from a variable to SQL. Then you can batch all in a single vs multiple individual queries...
vfpField = 28
vfpString = 'Smith'
text to lcSqlCmd noshow
select
YT.blah,
YT.blah2
into
#tempSqlResult
from
yourTable YT
where
YT.SomeKey = ?vfpField
select
ost.Xblah,
t.blah,
t.blah2
from
OtherSQLTable ost
join #tempSqlResult t
on ost.Xblah = t.blahKey;
drop table #tempSqlResult;
endtext
nHandle = sqlconnect( "your connection string" )
nAns = sqlexec( nHandle, lcSqlCmd, "LocalVFPCursorName" )
No I don't have error trapping in here, just to show principle and readability. I know the sample query could have easily been done via a join, but if you are working with some pre-aggregations and want to put them into temp work areas like Localized VFP cursors from a query to be used as your next step, this would work via #tempSqlResult as "#" indicates temporary table on SQL for whatever the current connection handle is.
If you want to return MULTIPLE RESULT SETs from a single SQL call, you can do that too, just add another query that doesn't have an "into #tmpSQLblah" context. Then, all instances of those result cursors will be brought back down to VFP based on the "LocalVFPCursorName" prefix. If you are returning 3 result sets, then VFP will have 3 cursors open called
LocalVFPCursorName
LocalVFPCursorName1
LocalVFPCursorName2
and will be based on the sequence of the queries in the SqlExec() call. But if you can provide more on what you ARE trying to do and their samples, we can offer more specific help too.
While this seems like a basic problem, I've been ripping my hair out FOR DAYS trying to get an efficient solution to this.
I have a lookup table of values on a server that I read from and assemble into a string using a C# Script task. I write this string into a variable that I want to pass in as my WHERE parameters inside a large SQL query on a ADO.NET data source (from a different server which I only have read access to) in my data flow. For example, this string would just be something like
('Frank', 'John', 'Markus', 'Tom')
and I would append that as my WHERE clause.
I can't read from a variable directly for an ADO.NET data source AND I can't use the 'Expression' property to set my SQL either as my SQL query is over 4000 characters. I could use an Execute SQL Task to run my query, load the results into a recordset and I assume, then loop through the recordset but that's extremely inefficient.
What would be the best way to do this? My end goal is to put these results inside a table on the first server.
You could try to set up Script Component as source - variables and strings inside scripts can be longer than 4000 characters so you can fit your query inside.
Setup your component similar to this article: http://beyondrelational.com/modules/2/blogs/106/posts/11119/script-componentsource-part1.aspx
In this one you have example how to fetch data using ExecuteReader and put it to output of script component: http://beyondrelational.com/modules/2/blogs/106/posts/11124/ssis-script-component-as-source-adonet.aspx In this one you have instructions how to aquire connection properly: http://www.toadworld.com/platforms/sql-server/b/weblog/archive/2011/05/30/use-connections-properly-in-an-ssis-script-task
By joining this pieces of information you should be able to write your source Script Component which can fetch data using any length dynamically constructed query.
Good luck :)
You can do a simple select statement to return a list of values that will include ('Frank', 'John', 'Markus', 'Tom'). So your select would return :
Name
----------
Frank
John
Markus
Tom
Then, in SSIS, use a Merge Join Component (that will act as a INNER JOIN) instead of a where clause in your main query.
This is the cleaniest way to achieve what you want.