In BigQuery, I can compose a query and then set a destination table under More > Query Settings. This works as expected for queries without variables, for example:
SELECT * FROM foo.bar WHERE PARSE_TIMESTAMP("%a, %d %b %Y %X %z", date_created) > '2020-01-01 00:00:00';
However, when I try to replace that formatting string with a variable, suddenly the options to set a destination table do not exist under More > Query Settings. For example:
DECLARE date_format STRING DEFAULT "%a, %d %b %Y %X %z";
SELECT * FROM foo.bar WHERE PARSE_TIMESTAMP(date_format, date_created) > '2020-01-01 00:00:00';
Additionally, even when I try to schedule the second query, I do not have an option to set a destination table.
Is this behavior expected? Is it documented anywhere? I have been unable to find an explanation.
It is not because of use of parameters per se!
But rather limitation of scripting.
So, YES, it is expected - when you use scripting you cannot use destination otherwise you will be getting error
If you need to get result into some table - just use INSERT INTO or any other relevant DML/DDL within your script
Related
I am trying to insert data from one BigQuery table to another using an Airflow DAG. I want to filter data such that the updateDate in my source table is greater than the previous execution date of my DAG run.
The updateDate in my source table looks like this: 2021-04-09T20:11:11Zand is of STRING data type whereasprev_execution_datelooks like this:2021-04-10T11:00:00+00:00which is why I am trying to convert myupdateDate` to TIMESTAMP first and then to ISO format as shown below.
SELECT *
FROM source_table
WHERE FORMAT_TIMESTAMP("%Y-%m-%dT%X%Ez", TIMESTAMP(UpdateDate)) > TIMESTAMP('{{ prev_execution_date }}')
But I am getting the error message: No matching signature for operator > for argument types: STRING, TIMESTAMP. Supported signature: ANY > ANY. Clearly the left hand side of my WHERE-clause above is of type STRING. How can I convert it to TIMESTAMP or to a correct format for that matter to be able to compare to prev_execution_date?
I have also tried with the following:
WHERE FORMAT_TIMESTAMP("%Y-%m-%dT%X%Ez", TIMESTAMP(UpdatedWhen)) > STRING('{{ prev_execution_date }}')
which results in the error message: Could not cast literal "2021-04-11T11:50:31.284349+00:00" to type DATE
I would appreciate some help regarding how to write my BigQuery SQL query to compare the String timestamp to previous execution date of Airflow DAG.
Probably you wanted to try parse_timestamp instead:
SELECT *
FROM source_table
WHERE PARSE_TIMESTAMP("%Y-%m-%dT%X%Ez", UpdateDate) > TIMESTAMP('{{ prev_execution_date }}')
although looks like it will work even without it
SELECT *
FROM source_table
WHERE TIMESTAMP(UpdateDate) > TIMESTAMP('{{ prev_execution_date }}')
I need to use a date/timestamp field in my where clause where it is >today()1pm
I'm setting it up to be a scheduled task, so the "today()" part is important
I dont know how to combine that with the time piece though
all I could come up with was this- and it seems like very inefficent code if it even works
datepart(fielda)=today() and timepart(fielda)>01:00:00pm
I need to be able to say here are the items with a timestamp of "today" after "1pm"
*I am passing my sql to an underlying database.
If you're using PROC SQL directly (not pass-through) you can use the SAS dhms function:
proc sql noprint;
* Assume myData contains a var named datestamp;
create table selected as
select * from myData
where datestamp > dhms(today(), 13, 0, 0)
;
quit;
The function constructs a datetime value from the supplied Date, Hour, Minutes and Seconds values, hence dhms.
This won't work if your SQL is being passed through to the underlying database because it's unlikely to understand the dhms function.
this is so much fancier and is easier to use
datefield between to_date(to_char(sysdate-1,'mm/dd/yyyy') || '12:00:00','mm/dd/yyyy hh24:mi:ss') and to_date(to_char(sysdate,'mm/dd/yyyy') || '08:32:27','mm/dd/yyyy hh24:mi:ss')
this will supply all values from yesterday at 12pm through 832am today
Combine a date9-formatted macro variable with the time. This will create a datetime literal. This method is going to be as efficient as can be, since SAS resolves the literal one time only, and can then use any existing index on datestamp.
%let today = %sysfunc(today(), date9.);
proc sql noprint;
create table selected as
select *
from mydata
where datestamp > "&TODAY.:13:00:00"dt
;
quit;
You can confirm the value with the code below.
%put %sysfunc(putn("&TODAY.:13:00:00"dt, mdyampm24.) );
For a SQL92 compliant database engine, the where syntax is as follows:
EXTRACT(HOUR FROM fielda) >= 13
The Query using LIKE :(This query when fired gives the desired result)
select * from catissue_audit_event where event_timestamp like '16-DEC-14'
But when using query with '=' results in an empty resultset
select * from catissue_audit_event where event_timestamp='16-DEC-14'
Here event_timestamp is of type Date
Strange thing is that the query runs for other dates such as:
select * from catissue_audit_event where event_timestamp='15-DEC-14'
What can be the issue? I already checked for leading and trailing spaces in the data
Output after running the first query:
In Oracle a DATE (and of course a TIMESTAMP) column contains a time part as well.
Just because your SQL client is hiding the time, doesn't mean it isn't there.
If you want all rows from a specific day (ignoring the time) you need to use trunc()
select *
from catissue_audit_event
where trunc(event_timestamp) = DATE '2014-12-16';
Be aware that this query will not use an index on the event_timestamp column.
You should also not rely on implicit data type conversion as you do with the expression event_timestamp = '16-DEC-14. That statement is going to fail if I run it from my computer because of different NLS settings. Always use a proper DATE literal (as I have done in my statement). If you don't like the unambiguous ISO date, then use to_date():
where trunc(event_timestamp) = to_date('16-12-2014', 'dd-mm-yyyy');
You should avoid using month names unless you know that all environments (which includes computers and SQL clients) where your SQL statement is executed are using the same NLS settings. If you are sure, you can use e.g. to_date('16-DEC-14', 'dd-mon-yy')
The reason why this is different is different to the solution to your issue.
The solution to your issue is to stop performing date comparisons by implicit conversion to a string. Convert your string to a date to perform a date comparison:
select * from catissue_audit_event where event_timestamp = date '2014-12-16'
I cannot stress this enough; when performing a date comparison only compare dates.
Your column EVENT_TIMESTAMP is being implicitly (this is bad) converted to a date in accordance with your NLS_DATE_FORMAT, which you can find as follows:
select * from nls_session_parameters
This governs how date-data is displayed and implicitly converted. The reason why LIKE works and and = doesn't is because your NLS_DATE_FORMAT is masking additional data. In other words, your date has a time component.
If you run the following and then re-select the data from your table you'll see the additional time component
alter session set nls_date_format = 'yyyy-mm-dd hh24:mi:ss'
Thus, if you want all the data for a specific date without constraint on time you'll need to remove the time component:
select * from catissue_audit_event where trunc(event_timestamp) = date '2014-12-16'
have you tried matching the event_timestamp format example: DD-MMM-YY with the date that you are passing?
I have a problem when try to select data from a table filtering by date.
For example:
SELECT * FROM MYTABLE WHERE MYTABLE.DATEIN = '23/04/49';
The Oracle Error is:
Informe de error:
Error SQL: ORA-01843: mes no válido
01843. 00000 - "not a valid month"
*Cause:
*Action:
Probably the source data of table is corrupted, in this case:
How can i solve this problem?
Can I change this dates for null?
The results of this select, select * from nls_session_parameters; , is:
PARAMETER VALUE
------------------------------ ----------------------------------------
NLS_LANGUAGE SPANISH
NLS_TERRITORY SPAIN
NLS_CURRENCY ¿
NLS_ISO_CURRENCY SPAIN
NLS_NUMERIC_CHARACTERS ,.
NLS_CALENDAR GREGORIAN
NLS_DATE_FORMAT DD/MM/RR
NLS_DATE_LANGUAGE SPANISH
NLS_SORT SPANISH
NLS_TIME_FORMAT HH24:MI:SSXFF
NLS_TIMESTAMP_FORMAT DD/MM/RR HH24:MI:SSXFF
NLS_TIME_TZ_FORMAT HH24:MI:SSXFF TZR
NLS_TIMESTAMP_TZ_FORMAT DD/MM/RR HH24:MI:SSXFF TZR
NLS_DUAL_CURRENCY ¿
NLS_COMP BINARY
NLS_LENGTH_SEMANTICS BYTE
NLS_NCHAR_CONV_EXCP FALSE
You should use the to_date function (oracle/functions/to_date.php
)
SELECT * FROM MYTABLE WHERE MYTABLE.DATEIN = TO_DATE('23/04/49', 'DD/MM/YY');
You are comparing a date column to a string literal. In such a case, Oracle attempts to convert your literal to a date, using the default date format.
It's a bad practice to rely on such a behavior, as this default may change if the DBA changes some configuration, Oracle breaks something in a future revision, etc.
Instead, you should always explicitly convert your literal to a date and state the format you're using:
SELECT * FROM MYTABLE WHERE MYTABLE.DATEIN = TO_DATE('23/04/49','MM/DD/YY');
If you don't need to check exact timestamp, use
SELECT * FROM MYTABLE WHERE trunc(DATEIN) = TO_DATE('23-04-49','DD-MM-YY');
otherwise, you can use
SELECT * FROM MYTABLE WHERE DATEIN = TO_DATE('23-04-49 20:18:07','DD-MM-YY HH24:MI:SS');
Here, you use hard code date,if you directly compare then you must use DD-MM-YY HH24:MI:SS else you might get ORA-01849: hour must be between 1 and 12.
I know this is a bit late, but I'm having a similar issue. SQL*Plus executes the query successfully, but Oracle SQL Developer shows the ORA-01843: not a valid month error.
SQL*Plus seems to know that the date I'm using is in the valid format, whereas Oracle SQL Developer needs to be told explicitly what format my date is in.
SQL*Plus statement:
select count(*) from some_table where DATE_TIME_CREATED < '09-12-23';
VS
Oracle SQL Developer statement:
select count(*) from some_table where DATE_TIME_CREATED < TO_DATE('09-12-23','RR-MM-DD');
Just in case this helps, I solved this by checking the server date format:
SELECT * FROM nls_session_parameters WHERE parameter = 'NLS_DATE_FORMAT';
then by using the following comparison (the left field is a date+time):
AND EV_DTTM >= ('01-DEC-16')
I was trying this with TO_DATE but kept getting an error. But when I matched my string with the NLS_DATE_FORMAT and removed TO_DATE, it worked...
In a comment to one of the answers you mention that to_date with a format doesn't help. In another comment you explain that the table is accessed via DBLINK.
So obviously the other system contains an invalid date that Oracle cannot accept. Fix this in the other dbms (or whatever you dblink to) and your query will work.
Having said this, I agree with the others: always use to_date with a format to convert a string literal to a date. Also never use only two digits for a year. For example '23/04/49' means 2049 in your system (format RR), but it confuses the reader (as you see from the answers suggesting a format with YY).
If the source date contains minutes and seconds part, your date comparison will fail.
you need to convert source date to the required format using to_char and the target date also.
If you are using command line tools, then you can also set it in the shell.
On linux, with a sh type shell, you can do for example:
export NLS_TIMESTAMP_FORMAT='DD/MON/RR HH24:MI:SSXFF'
Then you can use the command line tools and it will use the specified format:
/path/to/dbhome_1/bin/sqlldr user/pass#host:port/service control=table.ctl direct=true
Try using:
SELECT *
FROM MYTABLE
WHERE MYTABLE.DATEIN is not null
AND MYTABLE.DATEIN = '23/04/49';
Use the month as a string.
Example:
(12-Apr-2002) or (12-April-2002)
Although the answers using TO_DATE are correct, I prefer to use the ANSI SQL format for dates:
DATEIN = DATE '1949-04-23'
It works in Oracle and other DBMS ANSI SQL compliant. This is specially important if your application is DBMS agnostic.
Try alter session set NLS_DATE_FORMAT='DD/MM/YY'; -- or whatever format you want
I faced the same problem, on PROD, all the code were already in this format, but on preprod, it's not set,
So this means you change the default date format used by oracle
ALTER session set NLS_LANGUAGE=’AMERICAN’;
I have a table called users and In sql the format of date is yyyy-mm-dd
there fore when I try to enter data from my website in dd/mm/yyyy format it just enters 0000--00-00
How do I change the format it sql?
SQL Server Example
Link below has an easy explanation and there's an example if it helps
SELECT convert(varchar, getdate(), 103) – dd/mm/yyyy
Or try putting your column name in place of "datecolumn"
CONVERT(varchar(19), datecolumn, 103)
in SQL Server, according to the article here
SET DATEFORMAT YMD;
No Idea what data getting entered as 0000-00-00 and where zeros coming from , but this may help:
Basically , Data in a Date column in Oracle can be stored in any user defined format or kept as default.
It all depends on NLS parameter.
Current format can be seen by : SELECT SYSDATE FROM DUAL;
If you try to insert a record and insert statement is NOT in THIS format then it will give :
ORA-01843 : not a valid month error.
So first change the database date format before insert statements ( I am assuming you have bulk load of insert statements) and then execute insert script.
Format can be changed by :
ALTER SESSION SET nls_date_format = 'mm/dd/yyyy hh24:mi:ss';
Also You can Change NLS settings from SQL Developer GUI , (Tools > preference> database > NLS)
Ref: http://oracle.ittoolbox.com/groups/technical-functional/oracle-sql-l/how-to-view-current-date-format-1992815
If data is being saved but is showing up in your database as 0000-00-00 that'll usually be because you have the date field set to default at null.
Same thing happened to me. Once you turn that off it'll save the data as normal.
Worth also bearing in mind that you may need to ensure your code is being formatted properly when it reaches the database (if you happen to be posting data from a browser to the database. This will usually help to sort it:
$dateForDB = DateTime::createFromFormat('d-m-Y', $_POST['whateverYourDateVariableIsCalled']);
$dateForDB = $dateForDB->format('Y-m-d');
Hope that helps :)
(SELECT Top 1 REPLACE(CONVERT(CHAR(11), SendDate, 106),' ','-') FROM tbl_EmpDoc WHERE UniqueId=#UniqueId) AS SendDate