I'm calling a SQL query with a BigQuery API with Airflow. This query works perfectly fine in the BigQuery workspace but says I'm writing FROM FROM even though I'm not...
The logs say line 4, character 20 is where the error occurs which corresponds to:
, EXTRACT(DATE FROM event_time) AS session_date.
My overall query structure looks something like:
SELECT * FROM
((SELECT
fields_here
FROM table_name
LEFT JOIN UNNEST(sub_table) AS s
WHERE 1=1
UNION ALL
(SELECT
fields_here
FROM table_name
LEFT JOIN UNNEST(sub_table) AS s
WHERE 1=1
ORDER BY 1, 2))
ORDER BY 1, 2
I'm also using the LEAD() window function and COALESCE() but not sure if that matters. Really confused why this error is occurring...
Issue was not adding use_legacy_sql=False argument in Airflow
Related
Hey everyone im relativly new to Apache Drill and im having troubles converting my Oracle specific sql scripts (pl/sql) to Drill based querys.
For example i have a Scripts who checks for processed data in the last X Days.
In this script im using the the sysdate function.
Here is my old script:
SELECT i.id,i.status,status_text,i.kunnr,i.bukrs,i.belnr,i.gjahr,event,i.sndprn,i.createdate,executedate,tstamp,v.typ_text,i.docnum,i.description, i.*
FROM in_job i JOIN vstatus_injob v ON i.id= v.id
WHERE 1=1
AND i.createdate > sysdate - 30.5
order by i.createdate desc;
When i looked up in terms of drill specific Datetime Diff functions i found "TIMESTAMPDIFF".
So here is my "drillified" script:
SELECT i.id, i.status, status_text, i.kunnr, i.bukrs, i.belnr, i.gjahr, i.event, i.sndprn, i.createdate, i.executedate, i.tstamp,v.typ_text,i.docnum,i.description,i.*
FROM SchemaNAME.IN_JOB i JOIN SchemaNAME.VSTATUS_INJOB v ON i.id=v.id
WHERE TIMESTAMPDIFF(DAY, CURRENT_TIMESTAMP, i.createdate) >=30
And the Error that is returned reads like this:
DATA_READ ERROR: The JDBC storage plugin failed while trying setup the SQL query.
By further inspection i can see the Oracle specific error that reads:
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "TIMESTAMPDIFF": invalid ID
So now my question:
I thought apache drill replaces the function "TIMSTAMPDIFF" at runtime. But from what i can see in the logs its more like that Drill Hands over the Function Call "TIMESTAMPDIFF" to the Oracle database.
If thats true, how could i change my script to calculate the time difference (in days) and compare it to an int (ie 30 in the script).
If i use sysdate like above Apache Drill jumps in and says it doesnt know "sysdate".
How would you guyes handle that?
Thanks in advance and so long
:)
I have found a solution...
Just in Case someone (or even me in the future) is having a similar problem.
{
"queryType": "SQL",
"query": "select to_char(SELECT CURRENT_TIMESTAMP - INTERVAL XX MONTH FROM (VALUES(1)),'dd.MM.yy')"
}
With some to_char and the use of the CURRENT_TIMESTAMP - Interval Function Calls i can get everything i needed.
I took the query above packed it into an Grafana Variable, named it "timeStmpDiff" and then queried everything with an json Api Call to my Drill instance.
Basically:
"query" : "SELECT i.id, i.status, status_text, i.kunnr, i.bukrs, i.belnr, i.gjahr, i.event, i.sndprn, i.createdate, i.executedate, i.tstamp,v.typ_text,i.docnum,i.description,i.* FROM ${Schema}.IN_JOB i JOIN ${Schema}.VSTATUS_INJOB v ON i.id=v.id WHERE i.createdate >= '${timeStmpDiff}' order by i.createdate desc"
You can, of course query it in on go with an subselect.
But because i use grafana it made sense to me to bundle that in a Variable.
I am trying to make a query but google cloud gives a syntax error.
I had coppied this code which written in 2017 .
I have no idea about Sql
Syntax error: Unexpected "[" at [5:6]. If this is a table identifier, escape the name with `, e.g. `table.name` rather than [table.name].
The query is:
SELECT
f.repo_name,
f.path,
c.pkey
FROM
[bigquery-public-data:github_repos.files] f
JOIN (
SELECT
id,
You are probably using Standard SQL -- which is a good thing.
Try writing the table reference as:
FROM `bigquery-public-data.github_repos.files` f
I'm trying to use BigQuery's WITH clause as described in the documentation. I'm getting an error when running the following query:
WITH subQ1 AS (SELECT "1"),
subQ2 AS (SELECT "2")
SELECT * FROM subQ1
UNION ALL
SELECT * FROM subQ2;
The thrown error is:
Error: Encountered " "WITH" "WITH "" at line 1, column 1. Was expecting: <EOF>
Anyone has any idea of what I'm doing wrong?
You should do Enabling Standard SQL
Rather than this - you should be good!
BigQuery Legacy SQL does not support WITH
I'm trying to execute the following query in SQL Server, but it's throwing an error. How can I fix it?
select
T.T_Email
from
Stu_Question S, Tutor_Answer T
where
S.S_Quest_Id = '4f7a1518-a765-40c0-ae53-3ee61eef6673'
and S.S_Quest_Id = T.S_Quest_Id
and (T_Email,T_Answer_Update_Status)
IN (T_Email, Select MAX(T_Answer_Update_Status)
from Tutor_Answer
where S_Quest_Id='4f7a1518-a765-40c0-ae53-3ee61eef6673'
group by T_Email)
and S.S_Quest_Update_Status = (Select MAX(S_Quest_Update_Status)
from Stu_Question
where S_Quest_Id='4f7a1518-a765-40c0-ae53-3ee61eef6673')
This is the offending part of your statement:
and (T_Email,T_Answer_Update_Status)
IN (T_Email, Select MAX(T_Answer_Update_Status)
from Tutor_Answer
where S_Quest_Id='4f7a1518-a765-40c0-ae53-3ee61eef6673'
group by T_Email)
What on earth are you trying to do here???
T-SQL's IN operator works on one column at a time - like this:
WHERE T_EMail IN (SELECT EMail FROM .....)
marc_s correctly pointed out the offending part of your query.
You'll have to try to convert that to a join instead. Here is how I would do it:
select T.T_Email
from Stu_Question S
join (select T_Email,
row_number() over (partition by S_Quest_Id order by S_Quest_Update_Status desc) as rn
from Tutor_Answer) T
on T.S_Quest_Id=S.S_Quest_Id
and T.rn = 1
where S.S_Quest_Id='4f7a1518-a765-40c0-ae53-3ee61eef6673'
AND S.S_Quest_Update_Status=(Select MAX(S_Quest_Update_Status)
from Stu_Question
where S_Quest_Id='4f7a1518-a765-40c0-ae53-3ee61eef6673')
Notice that you can definitely improve this further. But it should get you going.
I am running a SQL query in stored procedure which is like following
SELECT
t1.id,t2.Name
FROM
table1 t1 , table2 t2 ,table2 t3,table4 t4
WHERE
t1.id=t3.t4.id
this query gets executed on SQL server 2008 when its compatible with SQL server 2000 but if we turn OFF the compatibility with SQL server 2000 then this Query gives syntax error which is expected.
Can some one help me to understand why this is happeneing ? thanks in advance
Original query:
SELECT
ConfigID , LocationDesc + '-' + LOBTeamDesc LocLOBTeamSource
FROM Config CONFIG , Location_LOBTeam LOCLOB , Location LOC , LOBTeam LOB, System SRC
WHERE CONFIG.LocationLOBTeamID = LOC.LOB.LocationLOBTeamID
AND CONFIG.SourceSystemID = SRC.SystemID
AND LOCLOB.LocationID = LOC.LocationID
AND LOCLOB.LOBTeamID = LOB.LOBTeamID
AND (GETDATE() BETWEEN CONFIG.effectiveDate AND CONFIG.EndDate
OR CONFIG.EndDate IS NULL)
ORDER BY
LOC.LocationCode
I think that original query, with current standard join syntax applied would be this:
SELECT
ConfigID
, LocationDesc + '-' + LOBTeamDesc LocLOBTeamSource
FROM Config CONFIG
INNER JOIN Location_LOBTeam LOCLOB
ON CONFIG.LocationLOBTeamID = LOCLOB.LocationLOBTeamID
INNER JOIN Location LOC
ON LOCLOB.LocationID = LOC.LocationID
INNER JOIN LOBTeam LOB
ON LOCLOB.LOBTeamID = LOB.LOBTeamID
INNER JOIN [System] SRC
ON CONFIG.SourceSystemID = SRC.SystemID
WHERE (GETDATE() BETWEEN CONFIG.effectiveDate AND CONFIG.EndDate
OR CONFIG.EndDate IS NULL)
ORDER BY
LOC.LocationCode
Perhaps this will help.
+EDIT
"System" as a table name, could that be a problem? Suggest you try it as [System]
+EDIT2
The original is given with this: LOC.LOB.LocationLOBTeamID but that appears to be an error as there is an alias LOCLOB
I think below post from msdn answers this issue Compatibility Levels and Stored Procedures
in the above post the point number 3 under section "Differences Between Compatibility Level 80 and Level 90" states "WHEN binding the column references in the ORDER BY list to the columns defined in the SELECT list, column ambiguities are ignored and column prefixes are sometimes ignored. This can cause the result set to return in an unexpected order."
on my database I am using compatibility level 80 i.e 2000 thats why it runs smoothly with the given syntax but when I remove this compatibility and make it to 100 i.e. 2008/R2 script gives syntax error which is expected