Related
Hey everyone im relativly new to Apache Drill and im having troubles converting my Oracle specific sql scripts (pl/sql) to Drill based querys.
For example i have a Scripts who checks for processed data in the last X Days.
In this script im using the the sysdate function.
Here is my old script:
SELECT i.id,i.status,status_text,i.kunnr,i.bukrs,i.belnr,i.gjahr,event,i.sndprn,i.createdate,executedate,tstamp,v.typ_text,i.docnum,i.description, i.*
FROM in_job i JOIN vstatus_injob v ON i.id= v.id
WHERE 1=1
AND i.createdate > sysdate - 30.5
order by i.createdate desc;
When i looked up in terms of drill specific Datetime Diff functions i found "TIMESTAMPDIFF".
So here is my "drillified" script:
SELECT i.id, i.status, status_text, i.kunnr, i.bukrs, i.belnr, i.gjahr, i.event, i.sndprn, i.createdate, i.executedate, i.tstamp,v.typ_text,i.docnum,i.description,i.*
FROM SchemaNAME.IN_JOB i JOIN SchemaNAME.VSTATUS_INJOB v ON i.id=v.id
WHERE TIMESTAMPDIFF(DAY, CURRENT_TIMESTAMP, i.createdate) >=30
And the Error that is returned reads like this:
DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query.
By further inspection i can see the Oracle specific error that reads:
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "TIMESTAMPDIFF": invalid ID
So now my question:
I thought apache drill replaces the function "TIMSTAMPDIFF" at runtime. But from what i can see in the logs its more like that Drill Hands over the Function Call "TIMESTAMPDIFF" to the Oracle database.
If thats true, how could i change my script to calculate the time difference (in days) and compare it to an int (ie 30 in the script).
If i use sysdate like above Apache Drill jumps in and says it doesnt know "sysdate".
How would you guyes handle that?
Thanks in advance and so long
:)
I have found a solution...
Just in Case someone (or even me in the future) is having a similar problem.
{
"queryType": "SQL",
"query": "select to_char(SELECT CURRENT_TIMESTAMP - INTERVAL XX MONTH FROM (VALUES(1)),'dd.MM.yy')"
}
With some to_char and the use of the CURRENT_TIMESTAMP - Interval Function Calls i can get everything i needed.
I took the query above packed it into an Grafana Variable, named it "timeStmpDiff" and then queried everything with an json Api Call to my Drill instance.
Basically:
"query" : "SELECT i.id, i.status, status_text, i.kunnr, i.bukrs, i.belnr, i.gjahr, i.event, i.sndprn, i.createdate, i.executedate, i.tstamp,v.typ_text,i.docnum,i.description,i.* FROM ${Schema}.IN_JOB i JOIN ${Schema}.VSTATUS_INJOB v ON i.id=v.id WHERE i.createdate >= '${timeStmpDiff}' order by i.createdate desc"
You can, of course query it in on go with an subselect.
But because i use grafana it made sense to me to bundle that in a Variable.
I am working on execute Script processor in nifi. I declared lastdate
as 2020-12-21 and I am trying to use this attribute in execute script
(groovy) file to fetch data from oracle.
In oracle it gave me the correct result. In nifi go to failure.
My code (script body):
def last_data = flowFile.getAttribute('last_date')
query = "select t.* from mytable where r.mydate > " + last_date + ",'yyyy-mm-dd')
Hehe, your variable is last_datA and you concatene last_datE.
I thinks that it buddy :)
I want to store current_day - 1 in a variable in Hive. I know there are already previous threads on this topic but the solutions provided there first recommends defining the variable outside hive in a shell environment and then using that variable inside Hive.
Storing result of query in hive variable
I first got the current_Date - 1 using
select date_sub(FROM_UNIXTIME(UNIX_TIMESTAMP(),'yyyy-MM-dd'),1);
Then i tried two approaches:
1. set date1 = ( select date_sub(FROM_UNIXTIME(UNIX_TIMESTAMP(),'yyyy-MM-dd'),1);
and
2. set hivevar:date1 = ( select date_sub(FROM_UNIXTIME(UNIX_TIMESTAMP(),'yyyy-MM-dd'),1);
Both the approaches are throwing an error:
"ParseException line 1:82 cannot recognize input near 'select' 'date_sub' '(' in expression specification"
When I printed (1) in place of yesterday's date the select query is saved in the variable. The (2) approach throws "{hivevar:dt_chk} is undefined
".
I am new to Hive, would appreciate any help. Thanks.
Hive doesn't support a straightforward way to store query result to variables.You have to use the shell option along with hiveconf.
date1 = $(hive -e "set hive.cli.print.header=false; select date_sub(from_unixtime(unix_timestamp(),'yyyy-MM-dd'),1);")
hive -hiveconf "date1"="$date1" -f hive_script.hql
Then in your script you can reference the newly created varaible date1
select '${hiveconf:date1}'
After lots of research, this is probably the best way to achieve setting a variable as an output of an SQL:
INSERT OVERWRITE LOCAL DIRECTORY '<home path>/config/date1'
select CONCAT('set hivevar:date1=',date_sub(FROM_UNIXTIME(UNIX_TIMESTAMP(),'yyyy-MM-dd'),1)) from <some table> limit 1;
source <home path>/config/date1/000000_0;
You will then be able to use ${date1} in your subsequent SQLs.
Here we had to use <some table> limit 1 as hive got a bug in insert overwrite if we don't specify a table name.
So I'm having trouble understanding on how to change the date on an update in postgres. What I have currently, that is giving a syntax error is
UPDATE works_locations SET (wrl_startdate = '2014-09-07', wrl_enddate = '2015-02-06')
with a few statements determining which field I should specifically change. However, postgres is giving me an error. How do I successfully change the date in postgres, even if the start date is around two years prior to this entry?
I don't have Postgres installed so I can't test this but try removing the parenthesis on your SET clause so that it looks like this:
UPDATE works_locations SET wrl_startdate = '2014-09-07', wrl_enddate = '2015-02-06'
I am using Oracle Client 10.2g and by changing my connection string to the Oracle database I now get error ORA00907 for some of my queries.
The code is executing within excel 2010 using VBA and I can run 20+ quires without error using the following connection string:
ServerConnectionString="Driver={Oracle in OraClient10g_home1};Dbq=DBNAME;Uid=USERNAME;Pwd=PASSWORD;"
With OpenR2DBConnection
.ConnectionTimeout = ConnectionTimeout
.Open ServerConnectionString
.Execute "ALTER SESSION SET NLS_DATE_FORMAT = 'DD/MM/YYYY'"
.CommandTimeout = CommandTimeout
End With
By changing the connection string only to:
ServerConnectionString="Provider=OraOLEDB.Oracle;Data Source=(<<<data exact translation from TNSNAMES file for the DBNAME>>>);User id=USERNAME;Password=PASSWORD;"
2 of the 20+ queries fail with ORA-00907: missing right parenthesis
One of the Queries that fails (refracted):
select n1.name,n1.sdate,n1.edate,n1.note as crnote,n1.cdate ,n1.pri
from
(
select n.name,n.sdate,n.edate,n.note,n.cdate,n.pri , RANK() OVER (PARTITION BY n.sdate,pri ORDER BY (n.adate - n.cdate) asc,n.pri asc) RANK
from
(
select mmpe.name,mmpe.sdate,mmpe.edate,mmpe.cannote as note,mmpe.cdate,mmip.crdate as adate,mp.pri
from mmpe ,mmmip mmip, mp
where <<clauses1>>
and mmpe.name in (<<list of strings>>)
and mmip.name(+) = mmpe.name
and <<more clauses2>>
union
Select mmip.name,mmip.sdate,greatest(<<formula>>) as edate , <<create note>> as note ,mmip.crdate as cdate, td.adate,mp.pri
From mi , mmip,td, mp
Where <<clauses3>>
and mi.name in (<<list of strings>>)
union all
Select mmip.name,mmip.sdate,mmip.edate as edate , <<create note>> as note ,mmip.crdate as cdate , td.adate,mp.pri
From mi , mmip,td, mp
Where <<clauses4>>
and mi.name in (<<list of strings>>)
union
Select mmip.name,mmip.sdate,td.cddate-1 as edate , <<create note>> as note, mmip.crdate as cdate,mmip.crdate as adate,mp.pri
From mi , mmip,td, mp
Where <<clauses4>>
and mi.name in (<<list of strings>>)
) n
) n1
where rank = 1
order by 6,2,5;
I have tested that the Query runs correctly in Oracle SQL Developer.
I have verified that prior to Executing the query the SQL statements are identical for both connection strings.
The other query that fails is also using a union and rank function but it is not the only one.
The reason I wish to use the OraOLEDB.Oracle connection is that I am attempting to remove my reliance on the tnsnames.ora files as from time to time I add new database instances and want to avoid all my users having to update this file in the oracle directory.
Lastly the ORACLE database is version 8.
Any help would be greatly appreciated,
Thanks in advance!
Updated: Removed typo error
OK I've work it out.
Thanks to all those that spent any time thinking about it.
When I refracted the example above I removed inline comments denoted by '--'.
It was these comments that caused the ORA-00907 error with the OraOLEDB.Oracle driver.
So the fix is simple: Remove comments from the SQL command!
Thanks again!