I am doing something that seems like it should be very easy yet I have yet to figure this out. I have read countless posts and tried everything I can think of and still no success.
Here goes:
I created a Pipeline Parameter pplLastWritten with a default value of 2022-08-20 12:19:08 (I have tried without the time for troubleshooting and still get errors)
Then I create a Data Flow Parameter ptblTableName
I have tried to convert to a Date, keeping as is and converting later...you name it still errors out.
In the expression builder I tried this and many more ways to build out to a sql statement:
"SELECT * FROM xxxxxx."+$ptblTableName+"where Lastwritten>='{$ptblLastWritten}'"
This is the post I got the idea from: ADF data flow concat expression with single quote
This is the error I got most of the time.
Operation on target df_DynamicSelect failed: {"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed due to reason: at Source 'RptDBTEST'(Line 5/Col 0): The store configuration is not defined. This error is potentially caused by invalid parameter assignment in the pipeline.","Details":""}
I have tried so many things but in the end nothing has worked. I am new to Data Factory and come from the SSIS world which was so much easier. I would greatly appreciate someone helping. Please explain this like I'm a kindergartener because this tool is making me feel like it. :) Thank you in advanced.
I have tried various ways to format
Using different ideas in the expression builder
the ideas in this post: ADF data flow concat expression with single quote
You can use concat() function in the Data flow dynamic expression like below.
Here is the sample data in SQL.
I have created two dataflow parameters mytable and mydate.
Passed the values like below. Check the expression checkbox. For date you can also pass like this '2022-11-07T00:00:00.0000000'.
In the Query option use below Expression.
concat('select * from dbo.',$table_name,' where mydate >=','\'',$mydate,'\'')
Values inserted in Target table.
Related
I am trying to build a fully parametrised pipeline template in ADF. With the work I have done so far, I can do a full load without any issues but when it comes to delta load, it seems like my queries are not working. I believe the reason for this is that my "where" statement looks somewhat like this:
SELECT #{item().source_columns} FROM #{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} > #{item().max_watermarkcolumn_loaded} AND #{item().source_watermarkcolumn} <= #{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}
where the 'max_watermarkcolumn_loaded' is a datetime format and the 'activity' output is obviously a string format.
Please correct me if my assumption is wrong and let me know what I can do to fix.
EDIT:
screenshot of the error
ADF is picking a date from SQL column 'max_watermarkcolumn_loaded' in this format '"2021-09-29T06:11:16.333Z"' and I think thats where the problem is.
I tried to repro this error. I gave the parameter without single quotes to a sample Query.
Wrap the date parameters with single quotes.
Corrected Query
SELECT #{item().source_columns} FROM
#{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} >
'#{item().max_watermarkcolumn_loaded}' AND
#{item().source_watermarkcolumn} <=
'#{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}'
With this query, pipeline is run successfully.
I'm a complete beginner with Azure Data Factory and I'm struggling to set a parameter using a lookup output, and then use the parameter within my query to import data.
I'm using an Azure SQL database as the source for the lookup and an Oracle database as the source for the copy data
My lookup query works ok and my copy data query works without a parameter so I know I'm part way there, but I'm struggling with the parameter part.
My lookup details are below with the output query
The copy data query is below with the lookup activity as the parameter added but as you can see it's not correct.
I'm not sure if I should be using the lookup as the parameter or if I should be using a parameter with the value generated from the lookup. In which case I have no idea how to do this.
I'm also pretty sure the parameter in the query needs some work as even hardcoding a parameter doesn't seem to work.
If anyone can help to get this working I'd be very grateful.
There doesn't seem to be much help with turning lookups into parameters on the internet.
Thanks in advance for any help anyone can give.
Try enclosing the lookup output as follows. Select column from table where rowstamp> '#{activity('Lookup1').output.firstrow.RowStamp}'
To proceed with a Database validation, I am having a need of comparing a record in the DB along with a data which is dynamically generated in the previous REST response, using SoapUI.
I have already captured the property value using a Property Transfer step and stored the required value in a custom property in Test-Case successfully i.e., using property expansion, say ${TestCase#customerId}
My intention is to use that particular value stored in the custom properties to query the result I am expecting, in the JDBC Request test step.
The query which I have drafted with the parameter is as below :
Select *
From ABC.SEC_CUST
Where ABC.SEC_CUST.CUSTOMER_ID = ${TestCase#customerId}
The response I receive after executing is as below.
Error getting response; java.sql.SQLSyntaxErrorException : ORA-00911: Invalid character.
But, when I run the query without the parameterized value it executes perfectly. Where, I tend the conclusion as there is a syntax issue in the way I have mentioned the parameter in the query.
But, I am unable to find the correct way to mention the parameter in the query in SoapUI.
Can anyone with experience in SoapUI, please assist me on this?
That is not working because of the use of property expansion which is only known to SoapUI, but not for the SQL query.
In order to get it work for the same, you need to define the variables in the top for all the parameters that are going to be used in the sql query.
Here the screen shot which explains how to use the same:
You forgot a '#'
Select *
From ABC.SEC_CUST
Where ABC.SEC_CUST.CUSTOMER_ID = ${#TestCase#customerId}
Try this.
Select * From ABC.SEC_CUST
Where ABC.SEC_CUST.CUSTOMER_ID = :customerId
I am using PDI to delete and insert some data from a DB. I have the following issue. I create two variables called START_DATE and END_DATE that are used to select the data that will be deleted from my DB. I am able to get them and run my transformation with no erors in the log file, but when I checked if data was deleted, I find it didn't. I send checked my "DeleteProcedure" step, and it says "Conversion error: null". I have tried different approached to take the variables and pass them as Strings, but I haven't been able to solve this issue. It cannot be a SQL mistake as I tested it with a constant and it works.
Any ideas? I attach some pics. Thanks!
As a documentation of the Execute SQL script says:
Note: When you have an issue, that the SQL is started at the initialization phase of the transformation and not for each row, make sure to check the option "Execute for each row" (see description below).
In your case it executes during the initialization phase of the transformation that's why it gets null values instead of ones from previous step.
I know, that similiar post was posted already (SQL query from Toplink expression), but I didnt find an answer there. I would like to get SQL query from Expression Builder expression:
I have
Expression exp = builder.get(NUMBER.getAttributeName()).equal(getNumber());
and I want to see the SELECT statement, like (Select * from table WHERE number=....)
Or is it possible to execute the expression from Expression Builder without session? (I know that when I used query.prepareCall(session, new DatabaseRow()) I can obtain statement, but I just need to avoid using session. Thank you very much.
You must have the Session (what are you going to execute it on without a Session???).
query.prepareCall(session, new DatabaseRow())
This is how you get the SQL, you need to create a ReadAllQuery with the expression to be able to get the SQL (an Expression is just a where clause).
The problem was, that i wasnt able to create session because I couldnt register my project. And that was because I miss one line of code in my Project.class
setName(APPLICATION_NAME);
After this, I was able to create session and execute query. Thank you anyway