I am new to Informatica Data Quality Analyst (Version 9.5.1 HotFix3) and I am having trouble in generating a basic SQL statement.
The SQL statement is being written against a mapping specification of a table that was originally imported as a flat file. The statement looks like:
Select ColumnA, ColumnB FROM Table1
WHERE Table1.ColumnA = 'S'
The SELECT .... FROM portion of the statement works fine but I encounter errors when I throw in the WHERE clause. I think my statement looks like standard SQL so I'm not sure why this will not work. Does Informatica Analyst accept SQL written only in a specific form? Are the inverted commas causing problems?
the query must work which you are trying to execute. If its not fetching the results, you need to do the following steps:
1) Load the data from source flat file to any database(oracle). You can directly import the data from flat file to table via sql developer.
2) Execute the query with the filter condition. If it doesnt fetch any rows, your query in idq is fetching correct results. If not, there is something missing in your idq code.
Related
I am having a an issue with SQL Command and CASE. I am pretty new to Crystal Reports/SQL and I have a basic code that I am playing around with to learn. I want to clean out a field -- that has imported from SQL Server. I just want to do something simple like this:
SELECT "I"."I_TYPE", "Alleg” =
CASE
WHEN "ALLEGs"."ALLEG” LIKE ‘*im*’ THEN ‘Improper’
ELSE ‘UNKNOWN’
END
I get an error that says Database Connector Error:
'42000:[MS][SQL..Incorrect syntax near ' . ' . Databse vender code 102.
Can you even use CASE as an IF THEN statement in the SQL Command. I am aware of SQL expressions, but I am trying to pull data to sql command to prevent performance decrease.
I am not sure about crystal report but your query formation don't look correct. It should be
SELECT I.I_TYPE,
CASE WHEN ALLEGs.ALLEG LIKE '%im%' THEN 'Improper' ELSE 'UNKNOWN' END AS 'Alleg'
Rahul, is correct for direct SQL commands to the server.
However sometimes when running Crystal Reports we use VBA within the report to do some data tuning rather than modifying the raw SQL on the fly.
This leaves the raw SQL a known result (verifiable on the SQL Server) and then modify the output in Crystal to fit the end users filtering requirements.
This is not efficient with large result sets but when the results are smaller (under 50k records) we usually have our team go with simple (post filtering) to reduce design and testing time.
This technique works very well with dynamic filters on the option section.
example: [Record Selection]
if {?Select Sales Person} <> "ENT" then
{R0033___P2A.ProjectionSP} = {?Select Sales Person} and {R0033___P2A.FM} >= 0
else
{R0033___P2A.ProjectionSP} > "" and {R0033___P2A.FM} >= 0
Where {?Select Sales Person} is a user selection filter and {R0033___P2A} is a predefined report view or stored procedure.
I am using pentaho for data migration testing. I have set a "table input" step where many parts of the query inside "table inputs" are variables. I have been looking for a way to capture that query after it gets executed during runtime.
I was wondering if there is any specific system log variables for sql or is it to do with metadata. need help! Thanks
Maybe the following approach will help:
We assume a transformation reading a CSV file to get the dynamic portion of the SELECT statement (e.g. the columns) and setting the variable columns with it.
The second transformation uses this variable to generate the SELECT statement and store it into the variable sql_statement.
In the main transformation we use ${sql_statement} as the SELECT statement of the table input and write the data to an output file (that's the business process so to say). From the same input we copy the output to another path. There we add the current time as a field (use element "Get system data") and we add the generated SQL statement, join them as a cartesian product and group the result by the sql_statement. That way we can compute the first time and the last time that the statement was used. These results are written to a text file.
The last thing we need is a job calling the three transformations sequentially.
This is a sample output:
sql_statement;min_time;max_time
SELECT my_column FROM test_table;2014/05/08 00:41:21.143;2014/05/08 00:41:21.144
Thank you Marcus! I did some thing similar.
It works. awesome.
I gathered parts of queries from table field where they were kept and formed a full query in javascript. After that full query will be sent as parameter to a transformation that will run and log the query.
While this seems like a basic problem, I've been ripping my hair out FOR DAYS trying to get an efficient solution to this.
I have a lookup table of values on a server that I read from and assemble into a string using a C# Script task. I write this string into a variable that I want to pass in as my WHERE parameters inside a large SQL query on a ADO.NET data source (from a different server which I only have read access to) in my data flow. For example, this string would just be something like
('Frank', 'John', 'Markus', 'Tom')
and I would append that as my WHERE clause.
I can't read from a variable directly for an ADO.NET data source AND I can't use the 'Expression' property to set my SQL either as my SQL query is over 4000 characters. I could use an Execute SQL Task to run my query, load the results into a recordset and I assume, then loop through the recordset but that's extremely inefficient.
What would be the best way to do this? My end goal is to put these results inside a table on the first server.
You could try to set up Script Component as source - variables and strings inside scripts can be longer than 4000 characters so you can fit your query inside.
Setup your component similar to this article: http://beyondrelational.com/modules/2/blogs/106/posts/11119/script-componentsource-part1.aspx
In this one you have example how to fetch data using ExecuteReader and put it to output of script component: http://beyondrelational.com/modules/2/blogs/106/posts/11124/ssis-script-component-as-source-adonet.aspx In this one you have instructions how to aquire connection properly: http://www.toadworld.com/platforms/sql-server/b/weblog/archive/2011/05/30/use-connections-properly-in-an-ssis-script-task
By joining this pieces of information you should be able to write your source Script Component which can fetch data using any length dynamically constructed query.
Good luck :)
You can do a simple select statement to return a list of values that will include ('Frank', 'John', 'Markus', 'Tom'). So your select would return :
Name
----------
Frank
John
Markus
Tom
Then, in SSIS, use a Merge Join Component (that will act as a INNER JOIN) instead of a where clause in your main query.
This is the cleaniest way to achieve what you want.
I have not had experience in SQL queries or SQL database , so please excuse me if my terminology is wrong.
So, I have a file containing around 17,000 SQL insert statements where I enter data for 5 columns/attributes in a database. In those 17,000 statements there are only around 1200 statements which have data for all of the 5 columns/attributes while the rest have data only for 4 columns. I need to delete all those unwanted statements( which dont have data for all 5 columns).
Is there a simple way/process to do it other than going one by one and deleting? If so, it would be great if someone could help me out with it.
A different approach from my fine colleagues here would be to run the file into a staging/disposable database. Use the delete that #Rob called out in his response to pare the table down to the desired dataset. Then use an excellent, free tool like SSMS Tools Pack to reverse engineer those insert statements.
I can think of two approaches:
1: Using SQL: insert all the data and then run a query that removes any records where it does not have all of the necessary data. If the table is not currently empty, keep track of the ID where your current data "ends" so that your query can use that as a WHERE statement.
DELETE FROM myTable WHERE a IS NULL OR b IS NULL /* etc. */
2: Process the SQL file with a regular expression: Use a text editor or command line to match either "bad" records or "good" records. Most text editors have a find and replace that allows you to use regular expressions. And command line you can use grep or other tools to process. Or even a script that parses in your language of choice, for that matter.
Open file in notepad++, replace all "bad" lines using regular expressions.
The only thing I don't have an automated tool for when working with Oracle is a program that can create INSERT INTO scripts.
I don't desperately need it so I'm not going to spend money on it. I'm just wondering if there is anything out there that can be used to generate INSERT INTO scripts given an existing database without spending lots of money.
I've searched through Oracle with no luck in finding such a feature.
It exists in PL/SQL Developer, but errors for BLOB fields.
Oracle's free SQL Developer will do this:
http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html
You just find your table, right-click on it and choose Export Data->Insert
This will give you a file with your insert statements. You can also export the data in SQL Loader format as well.
You can do that in PL/SQL Developer v10.
1. Click on Table that you want to generate script for.
2. Click Export data.
3. Check if table is selected that you want to export data for.
4. Click on SQL inserts tab.
5. Add where clause if you don't need the whole table.
6. Select file where you will find your SQL script.
7. Click export.
Use a SQL function (I'm the author):
https://github.com/teopost/oracle-scripts/blob/master/fn_gen_inserts.sql
Usage:
select fn_gen_inserts('select * from tablename', 'p_new_owner_name', 'p_new_table_name')
from dual;
where:
p_sql – dynamic query which will be used to export metadata rows
p_new_owner_name – owner name which will be used for generated INSERT
p_new_table_name – table name which will be used for generated INSERT
p_sql in this sample is 'select * from tablename'
You can find original source code here:
http://dbaora.com/oracle-generate-rows-as-insert-statements-from-table-view-using-plsql/
Ashish Kumar's script generates individually usable insert statements instead of a SQL block, but supports fewer datatypes.
I have been searching for a solution for this and found it today. Here is how you can do it.
Open Oracle SQL Developer Query Builder
Run the query
Right click on result set and export
http://i.stack.imgur.com/lJp9P.png
You might execute something like this in the database:
select "insert into targettable(field1, field2, ...) values(" || field1 || ", " || field2 || ... || ");"
from targettable;
Something more sophisticated is here.
If you have an empty table the Export method won't work. As a workaround. I used the Table View of Oracle SQL Developer. and clicked on Columns. Sorted by Nullable so NO was on top. And then selected these non nullable values using shift + select for the range.
This allowed me to do one base insert. So that Export could prepare a proper all columns insert.
If you have to load a lot of data into tables on a regular basis, check out SQL Loader or external tables. Should be much faster than individual Inserts.
You can also use MyGeneration (free tool) to write your own sql generated scripts. There is a "insert into" script for SQL Server included in MyGeneration, which can be easily changed to run under Oracle.