converting a DateTime column to string in ADF - sql

I am trying to build a fully parametrised pipeline template in ADF. With the work I have done so far, I can do a full load without any issues but when it comes to delta load, it seems like my queries are not working. I believe the reason for this is that my "where" statement looks somewhat like this:
SELECT #{item().source_columns} FROM #{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} > #{item().max_watermarkcolumn_loaded} AND #{item().source_watermarkcolumn} <= #{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}
where the 'max_watermarkcolumn_loaded' is a datetime format and the 'activity' output is obviously a string format.
Please correct me if my assumption is wrong and let me know what I can do to fix.
EDIT:
screenshot of the error
ADF is picking a date from SQL column 'max_watermarkcolumn_loaded' in this format '"2021-09-29T06:11:16.333Z"' and I think thats where the problem is.

I tried to repro this error. I gave the parameter without single quotes to a sample Query.
Wrap the date parameters with single quotes.
Corrected Query
SELECT #{item().source_columns} FROM
#{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} >
'#{item().max_watermarkcolumn_loaded}' AND
#{item().source_watermarkcolumn} <=
'#{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}'
With this query, pipeline is run successfully.

Related

How to pass a Date Pipeline Parameter to a Data Flow use in a Dataflow Expression Builder

I am doing something that seems like it should be very easy yet I have yet to figure this out. I have read countless posts and tried everything I can think of and still no success.
Here goes:
I created a Pipeline Parameter pplLastWritten with a default value of 2022-08-20 12:19:08 (I have tried without the time for troubleshooting and still get errors)
Then I create a Data Flow Parameter ptblTableName
I have tried to convert to a Date, keeping as is and converting later...you name it still errors out.
In the expression builder I tried this and many more ways to build out to a sql statement:
"SELECT * FROM xxxxxx."+$ptblTableName+"where Lastwritten>='{$ptblLastWritten}'"
This is the post I got the idea from: ADF data flow concat expression with single quote
This is the error I got most of the time.
Operation on target df_DynamicSelect failed: {"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed due to reason: at Source 'RptDBTEST'(Line 5/Col 0): The store configuration is not defined. This error is potentially caused by invalid parameter assignment in the pipeline.","Details":""}
I have tried so many things but in the end nothing has worked. I am new to Data Factory and come from the SSIS world which was so much easier. I would greatly appreciate someone helping. Please explain this like I'm a kindergartener because this tool is making me feel like it. :) Thank you in advanced.
I have tried various ways to format
Using different ideas in the expression builder
the ideas in this post: ADF data flow concat expression with single quote
You can use concat() function in the Data flow dynamic expression like below.
Here is the sample data in SQL.
I have created two dataflow parameters mytable and mydate.
Passed the values like below. Check the expression checkbox. For date you can also pass like this '2022-11-07T00:00:00.0000000'.
In the Query option use below Expression.
concat('select * from dbo.',$table_name,' where mydate >=','\'',$mydate,'\'')
Values inserted in Target table.

Pulling a SQL Server date to be used in Oracle query within SSIS environment is ignored in the Oracle query

I am having trouble in my Oracle query that uses a variable stored in SSIS which has a date that is pulled from sql server.
I am using an execute sql task that simply gets a max date from a sql server table and stores it in a variable. E.g.
SELECT MAX(t.Date) FROM table t;
I then want to use that variable in my Oracle query which is an ADO.NET source connection. I noticed you can't parameterize in those connections and found the work around where you use the sql expression with your user variable in it. So now my Oracle source query looks something like this:
"SELECT DISTINCT t.* FROM table t WHERE TO_CHAR(t.LastUpdateDate, 'YYYY-MM-DD') > " + "'#[User::LastUpdateDate]'"
The query syntax itself is fine, but when I run it, it is pulling all rows and seems to be completely ignoring the where clause of the date.
I've tried removing the TO_CHAR from LastUpdateDate.
I've tried adding a TO_CHAR to my user variable #[User::LastUpdateDate].
I've tried using the CONVERSION() function from sql server on #[User::LastUpdateDate].
Nothing seems to work and the query just runs and pulls in all data as if I don't have the WHERE clause on the query.
Does anyone know how to rectify this issue or point out what I might be doing wrong?
Thank you for any and all help!
**EDIT:
My date being pulled from SQL Server is in this format: 2022-09-01 20:17:58.0000000
This is not an answer, just troubleshooting advice
You do not say what data type #[User::LastUpdateDate] is, I'll assume it's a datetime
Ideally all datetime data should be kept in datetime data types, then format becomes completely irrelevant. However since it's difficult to parameterise Oracle queries in SSIS, you have to concoct a string to be submitted. Now date format does become important.
On to something a little different, it is a very good habit performancewise, to not put functions around columns that you are searching on. This is called sargability - look it up.
Given these things, I suggest that you concoct your required SQL query bit by bit and troubleshoot.
First, format your date parameter as an Oracle date literal. Remember this is normally a bad and unecessary thing. We are only doing it because we have to concoct a SQL string.
So create another SSIS variable called strLastUpdateDate and put this hideous expression in it:
RIGHT("0" + (DT_STR,2,1252)DATEPART( "dd" , #[User::LastUpdateDate] ), 2) + '-' +
(DT_STR,3,1252)DATEPART( "mmm" , #[User::LastUpdateDate] ) + '-' +
(DT_STR,4,1252)DATEPART("yyyy" , #[User::LastUpdateDate] )
Yes this is ludicrously long code but it will turn your date variable into a Oracle string literal. You could simplify this by putting it into your original max query but lets not go there. Use whatever debugging technique you have to confirm that it works as expected.
Now you should be able to use this:
"SELECT t.*, '"+#[User::LastUpdateDate]+"' As MyStrDate FROM table t WHERE
t.LastUpdateDate > '" #[User::strLastUpdateDate] + "'"
You can try running that and see if it makes any difference. Make sure you use this https://dba.stackexchange.com/questions/8828/how-do-you-show-sql-executing-on-an-oracle-database to monitor what is actually being submitted to Oracle.
This is all from memory and googling - I haven't done SSIS for many years now
I suspect after all this you may still have the same problem because I recall from many years having the same mysterious issue.

SQL Runner (Google Looker) change date format in query

This topic has been covered several times but I can't find a solution that applies to SQL Runner, which is the custom query portion of Google's Looker platform.
I am attempting to reformat a datetime SELECT statement from yyyy-mm-dd to mm-dd-yyyy.
Currently what I have is:
SELECT
CAST(shift.datetime AS DATE)
FROM table.a
This gives me the yyyy-mm-dd result but so far my efforts to CONVERT have been fruitless. It does not appear that SQL Runner supports the CONVERT command or I am utilizing it incorrectly.
Any thoughts on this one?
I believe sql runner is just gives us a way to directly access the db and it will not change any sql query while communicating with the db directly as long as the timezone of both explore as well as db matches.
Maybe something like this should work for your case
https://sql.tutorialink.com/convert-yyyymmdd-to-mm-dd-yyyy-in-snowflake/
lmk if the above works for your or not!

DateAdd not working in Calculated Query Field

I have tried to make a calculated field in a query with the build tool
EindDatum: DateAdd("yyyy",1,[tblVERHURING]![DatumVerhuring])
I get an error
The expression you entered contains invalid syntax
If I use the SQL tool and type the following, I get the result I want
SELECT DATEADD("yyyy", 1, [tblVERHURING]![DatumVerhuring]) AS EindDatum
FROM tblVerhuring;
My question: is it possible to build this SQL expression with the build tool, and what should the code be?
you probably need to change the yyyy for your language.
Maybe this will help you:
EindDatum: DateAdd("jjjj",1,[tblVERHURING]![DatumVerhuring])
This works for Dutch versions of MS Access
see the related docs

Querying a pass through query Acess

I'm looking how to run a local query with date filters on a saved pass-through queries in Access. I'm trying to leverage the pass through query as basically a View in my Access database... the local query is constructed dynamically in VBA, and intended to be a clean way to filter my pass through query. I'd like to avoid creating another pass through query, or altering the original, every time i run my Sub Procedure.
My problem is that the normal access date filter format #m/d/yyyy# doesn't seem to work. I've tried both altering the date format as well in the pass through query with 1. Convert(varchar(12),p.startDate,101); 2. Convert(date,p.StartDate,101);
but neither will work when the pass through query is queried against locally.
Does anyone know how to do this?
UPDATE - I just checked and Access is reading the field as Text... does anyone know how it can read it as a date? As i mentioned the CONVERT functions don't seem to be working to do this
In a passthru you MUST use the backend's syntax. If the BE is SQL Server then I'd use a syntax like this:
WHERE DocDate = '2015-03-17'