I came across a surprising behaviour in Legacy SQL. Indeed, I launched this query by accident (I replace the '[[date]]' programmatically, I just forgot to replace it in UI) :
SELECT DATE(ComputationDate) as date
FROM [project:dataset.table]
WHERE DATE(ComputationDate) < '[[date]]'
ORDER BY date
And it worked ! I retrieved all data to today date. This fails in Standard SQL but was it an intended behaviour ?
It is not that much an issue but if my replacement fails, I cannot see it from code since the query still compiles.
Thanks
Below version (BigQuery Legacy SQL) will return no rows at all which can be good indicator for you that something went wrong
#legacySQL
SELECT DATE(ComputationDate) as date
FROM [project:dataset.table]
WHERE DATE(ComputationDate) < DATE('[[date]]')
ORDER BY date
The "culprit" is how it is cast under the hood. In my test, your query will return all the data for any string that cannot be parsed to a date type.
Related
I am trying to build a fully parametrised pipeline template in ADF. With the work I have done so far, I can do a full load without any issues but when it comes to delta load, it seems like my queries are not working. I believe the reason for this is that my "where" statement looks somewhat like this:
SELECT #{item().source_columns} FROM #{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} > #{item().max_watermarkcolumn_loaded} AND #{item().source_watermarkcolumn} <= #{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}
where the 'max_watermarkcolumn_loaded' is a datetime format and the 'activity' output is obviously a string format.
Please correct me if my assumption is wrong and let me know what I can do to fix.
EDIT:
screenshot of the error
ADF is picking a date from SQL column 'max_watermarkcolumn_loaded' in this format '"2021-09-29T06:11:16.333Z"' and I think thats where the problem is.
I tried to repro this error. I gave the parameter without single quotes to a sample Query.
Wrap the date parameters with single quotes.
Corrected Query
SELECT #{item().source_columns} FROM
#{item().source_schema}.#{item().source_table}
WHERE #{item().source_watermarkcolumn} >
'#{item().max_watermarkcolumn_loaded}' AND
#{item().source_watermarkcolumn} <=
'#{activity('Watermarkvalue').output.firstRow.max_watermarkcolumn_loaded}'
With this query, pipeline is run successfully.
I am having trouble in my Oracle query that uses a variable stored in SSIS which has a date that is pulled from sql server.
I am using an execute sql task that simply gets a max date from a sql server table and stores it in a variable. E.g.
SELECT MAX(t.Date) FROM table t;
I then want to use that variable in my Oracle query which is an ADO.NET source connection. I noticed you can't parameterize in those connections and found the work around where you use the sql expression with your user variable in it. So now my Oracle source query looks something like this:
"SELECT DISTINCT t.* FROM table t WHERE TO_CHAR(t.LastUpdateDate, 'YYYY-MM-DD') > " + "'#[User::LastUpdateDate]'"
The query syntax itself is fine, but when I run it, it is pulling all rows and seems to be completely ignoring the where clause of the date.
I've tried removing the TO_CHAR from LastUpdateDate.
I've tried adding a TO_CHAR to my user variable #[User::LastUpdateDate].
I've tried using the CONVERSION() function from sql server on #[User::LastUpdateDate].
Nothing seems to work and the query just runs and pulls in all data as if I don't have the WHERE clause on the query.
Does anyone know how to rectify this issue or point out what I might be doing wrong?
Thank you for any and all help!
**EDIT:
My date being pulled from SQL Server is in this format: 2022-09-01 20:17:58.0000000
This is not an answer, just troubleshooting advice
You do not say what data type #[User::LastUpdateDate] is, I'll assume it's a datetime
Ideally all datetime data should be kept in datetime data types, then format becomes completely irrelevant. However since it's difficult to parameterise Oracle queries in SSIS, you have to concoct a string to be submitted. Now date format does become important.
On to something a little different, it is a very good habit performancewise, to not put functions around columns that you are searching on. This is called sargability - look it up.
Given these things, I suggest that you concoct your required SQL query bit by bit and troubleshoot.
First, format your date parameter as an Oracle date literal. Remember this is normally a bad and unecessary thing. We are only doing it because we have to concoct a SQL string.
So create another SSIS variable called strLastUpdateDate and put this hideous expression in it:
RIGHT("0" + (DT_STR,2,1252)DATEPART( "dd" , #[User::LastUpdateDate] ), 2) + '-' +
(DT_STR,3,1252)DATEPART( "mmm" , #[User::LastUpdateDate] ) + '-' +
(DT_STR,4,1252)DATEPART("yyyy" , #[User::LastUpdateDate] )
Yes this is ludicrously long code but it will turn your date variable into a Oracle string literal. You could simplify this by putting it into your original max query but lets not go there. Use whatever debugging technique you have to confirm that it works as expected.
Now you should be able to use this:
"SELECT t.*, '"+#[User::LastUpdateDate]+"' As MyStrDate FROM table t WHERE
t.LastUpdateDate > '" #[User::strLastUpdateDate] + "'"
You can try running that and see if it makes any difference. Make sure you use this https://dba.stackexchange.com/questions/8828/how-do-you-show-sql-executing-on-an-oracle-database to monitor what is actually being submitted to Oracle.
This is all from memory and googling - I haven't done SSIS for many years now
I suspect after all this you may still have the same problem because I recall from many years having the same mysterious issue.
This topic has been covered several times but I can't find a solution that applies to SQL Runner, which is the custom query portion of Google's Looker platform.
I am attempting to reformat a datetime SELECT statement from yyyy-mm-dd to mm-dd-yyyy.
Currently what I have is:
SELECT
CAST(shift.datetime AS DATE)
FROM table.a
This gives me the yyyy-mm-dd result but so far my efforts to CONVERT have been fruitless. It does not appear that SQL Runner supports the CONVERT command or I am utilizing it incorrectly.
Any thoughts on this one?
I believe sql runner is just gives us a way to directly access the db and it will not change any sql query while communicating with the db directly as long as the timezone of both explore as well as db matches.
Maybe something like this should work for your case
https://sql.tutorialink.com/convert-yyyymmdd-to-mm-dd-yyyy-in-snowflake/
lmk if the above works for your or not!
I have an open query I am using to connect to a PROGRESS ODBC, I cannot get the syntax equivalent from SQL of
SELECT DATEADD(DAY, -1, GETDATE())
The Progress SQL documentation is here:
https://documentation.progress.com/output/OpenEdge117/openedge117/#page/dmsrf%2Fadd-months.html%23wwID0EGGFR
There is no DATEADD() function although there is ADD_MONTHS(). I'm not much of a SQL coder but I expect that you could probably cobble something together with the other date related functions.
I came up with the following to get records where the date is greater than two days prior:
where tr_date =
to_date({ fn convert (month(curdate()), sql_varchar)} + '/' + { fn convert(dayofmonth(curdate()) -2, sql_varchar)} + '/' + { fn convert (year(curdate()), sql_varchar)})
It heavily relies on curdate() which you apparently can't use in a select statement, only in a where clause.
In SQL Server I would have done all of this in front of the query, put the results in a variable and then used the variable in the query's where clause, but Progress also apparently doesn't have the concept of variables.
The problem with the above is that it doesn't account for the beginning of the month where substracting 2 from 1 would result in -1 and therefore an invalid date. So the algorithm to account for this becomes more complicated. All more doable if you had variables but trickier when you have to do it all on the fly.
My purpose was to generate a new query each day for an incremental data integration process using SSIS. I switched over to dynamically generating the query in SQL Server and inserting the results into an SSIS configuration table that the package reads. With that method you end up using good ol' reliable dateadd in SQL Server. That unfortunately doesn't help much if you're trying to do this in Progress with SQL there.
I'm using Visual Studio 2013, I try to use some function (like "cast" and Year ) in Table Adapter Query Builder (in DataSet.XSD ). I get erroe messgae every time. I run the sql statement on other sql programs and it's work fine. did anyone face this problem?
Is your dataSource SQL Server or SQLite. If you are using SqLite then functions such as Year(),Cast() are not allowed.
If you are using SQLite then please find below link for date time function reference,
https://www.sqlite.org/lang_datefunc.html
As you have asked for Cast function, there is SO post describing the cast function, which is similar to that of SQL Server
SQLite supports CAST and:
Casting an INTEGER or REAL value into TEXT renders the value as if via sqlite3_snprintf() except that the resulting TEXT uses the
encoding of the database connection.
So you can do things like this:
select cast(some_integer_column as text) from some_table;
Or, depending on what you're trying to do, you could just treat the
numbers as strings and let SQLite coerce the types as it sees fit:
select some_int || ' pancakes' from some_table; select some_int || ''
from some_table;
SQLite does not have this function YEAR(...). Try strftime('%Y', degrees.ExamDate) = '2017' instead.