Following is the query running on hive server 4 map reduce jobs are running . It fails after running the second map-reduce job. Belowi is the query and following the error that seems to be coming . Please let me know if you have any idea.
select kpl.normalized_keyword, kpl.ptitle_id as feature_id, kpl.tagcount as tag_count, kpl.market_id, kpl.rpd_metric, kpl.rpi_metric,
kpl.imps_day, kpl.clicks_day, kpl.clicks, kpl.lang
from
(select coalesce(kpl_sprf.normalized_keyword, kpl_sr.normalized_keyword) normalized_keyword, coalesce(kpl_sprf.ptitle_id, kpl_sr.ptitle_id) ptitle_id,
coalesce(kpl_sprf.tagcount, kpl_sr.tagcount) tagcount, coalesce(kpl_sprf.market_id, kpl_sr.market_id)
market_id, kpl_sprf.rpd_metric, kpl_sprf.rpi_metric, kpl_sprf.lang, kpl_sprf.imps_day, kpl_sprf.clicks_day, kpl_sr.clicks
from
(select kpl_sprf.normalized_keyword, kpl_sprf.lang, kpl_sprf.ptitle_id, kpl_sprf.tagcount, kpl_sprf.market_id, kpl_sprf.rpd_metric,
kpl_sprf.rpi_metric, case when kp.avg_imps_short is not null then kp.avg_imps_short/kp.interval_days_short when datediff(kp.last_modified,
kp.interval_start_short) <= 0 then kp.cur_imps_short/1.0 else kp.cur_imps_short/datediff(kp.last_modified, kp.interval_start_short) end as
imps_day, case when kp.avg_clicks_short is not null then kp.avg_clicks_short/kp.interval_days_short when datediff(kp.last_modified, kp.interval_start_short)
<= 0 then cur_clicks_short/1.0 else cur_clicks_short/datediff(kp.last_modified, kp.interval_start_short) end as clicks_day
from
kpl_sprf_wk kpl_sprf
left semi join
(select * from scfg_kpl_sprf_wk where date_ = '2015-02-26') kw on kw.normalized_keyword
= kpl_sprf.normalized_keyword and kw.lang = kpl_sprf.lang join (select * from sprf_keyword_ptitle_ng_stats_wk where date_ = '2015-02-26')
kp on kp.normalized_keyword = kpl_sprf.normalized_keyword and kp.ptitle_id = kpl_sprf.ptitle_id and kp.market_id = kpl_sprf.market_id where
(kp.cur_clicks_long >= 3 or kp.avg_clicks_long >= 3) and (datediff(kp.last_modified, kp.interval_start_long) >= 7 or
kp.avg_imps_long is not null) and kp.ptitle_id not between 1100000000 and 1100100000) kpl_sprf full outer join
(select kpl_sr.normalized_keyword, kpl_sr.ptitle_id, kpl_sr.tagcount, kpl_sr.market_id, kpl_sr.clicks from kpl_sr_wk kpl_sr left semi join
(select * from scfg_kpl_sprf_wk where date_='2015-02-26') kw on kw.normalized_keyword = kpl_sr.normalized_keyword and kw.lang = kpl_sr.lang )
kpl_sr on kpl_sr.normalized_keyword = kpl_sprf.normalized_keyword and kpl_sr.ptitle_id = kpl_sprf.ptitle_id and kpl_sr.market_id = kpl_sprf.market_id)
kpl where kpl.market_id in (1)
which is giving the following error
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"normalized_keyword":"","lang":"es","date_":"2015-02-26"}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:159)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"normalized_keyword":"","lang":"es","date_":"2015-02-26"}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:675)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:141)
... 8 more
Caused by: java.lang.RuntimeException: cannot find field normalized_keyword from [0:_col0, 1:_col1]
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:346)
at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:128)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:85)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:658)
This issue seems to be coming since a lot. I think it is because of join query used Any Idea how to resolve it ?
Somewhere in that query you are referencing the field normalized_keyword, but it wasn't found in the fields you had selected. First thing I would suggest is to replace SELECT * with the actual fields and work through each instance of normalized_keyword to make sure it is available to be selected.
Related
I have executed this query on HIVE, using that dataset:
https://data.europa.eu/data/datasets/erasmus-mobility-statistics-2014-2019-v2?locale=en:
select i.foe, i.pn, i.massimo
from
(
SELECT m.foe, m.pn, max(m.somma) AS massimo
FROM
(
select pn, foe, sum(participants) as somma
from erasmus
WHERE foe <> "? Unknown ?" and participants>0
group by foe, pn
) as m
group by m.foe, m.pn
) as i
where i.massimo=
(
select sum(participants) as somma
from erasmus
WHERE foe <> "? Unknown ?" and participants>0 AND i.foe=foe
group by foe, pn
)
At the end of execution I get this error back:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"_col0":"Audio-visual techniques and media production","_col1":"AD"},"value":null}
caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Scalar subquery expression returns more than one row.
How can I solve this? Thank you.
I'm trying to run a query in Hive where I run a join based on a case statement. For some reason, I'm having problems on lines 7 and 8. I have not been able to resolve the error which is
line 7: Expected: AND, AS, BETWEEN, DIV, ILIKE, IN, IREGEXP, IS, LIKE, NOT, OR, REGEXP, RLIKE CAUSED BY: Exception: Syntax error
line 8: Encountered: AS Expected: AND, BETWEEN, DIV, ILIKE, IN, IREGEXP, IS, LIKE, NOT, OR, REGEXP, RLIKE CAUSED BY: Exception: Syntax error
select * from dra_record_set.mark_set inv
INNER JOIN innerdb.name_set roll_table
on inv.record_id = roll_table.ply_record_id AND
roll_table.date =
(CASE
WHEN inv.purchase_day>0 AND inv.purhcase_date BETWEEN roll_table.discount_start_dt AND roll_table.discount_end_dt) THEN inv.purchase_date
ELSE WHEN (CONCAT(inv.purchase_yr,inv.purchase_mo,(CAST("15")AS INT))) AS temp_var BETWEEN roll_table.discount_start_dt AND roll_table.discount_end_dt) THEN inv.purchase_date
END AS temp_pur_dt)
WHERE inv.inroll_discount_eligible_flag =1
limit 10
case is an expression. It returns a value. Your seems to be more like a macro substitution for SQL code.
Remove the case and just use boolean logic:
from dra_record_set.mark_set inv join
innerdb.name_set roll_table
on inv.record_id = roll_table.ply_record_id and
( (inv.purchase_day > 0 and
inv.purhcase_date between roll_table.discount_start_dt and roll_table.discount_end_dt
) or
(inv.purchase_day <= 0 and
CONCAT(inv.purchase_yr, inv.purchase_mo, 15) between roll_table.discount_start_dt AND roll_table.discount_end_dt
)
)
I´ve been trying to find the error in this statement for a few hours and can´t seem to find it.
It must have something to do with the AND connecting the two WHERE´s since when deleting the first WHERE it works:
SELECT E_AUFMASS_KOMMENTARE.FIBU_FIRMA,
E_AUFMASS_KOMMENTARE.AUFTR_NR,
E_AUFMASS_KOMMENTARE.KOMMENTAR,
AUFTR_EXT.ART_GRUPPE
FROM HHNG_AU.E_AUFMASS_KOMMENTARE
INNER JOIN HHNG_AU.AUFTR_EXT ON E_AUFMASS_KOMMENTARE.AUFTR_NR = AUFTR_EXT.AUFTR_NR
WHERE (E_AUFMASS_KOMMENTARE.AUFTR_NR = '1248823' )
AND WHERE NOT EXISTS( SELECT * FROM HHNG_AU.EX_KOMMENTARE WHERE EX_KOMMENTARE.AUFTR_NR = '1248823' )
Too many WHERE. You only need where once, then use ANDs and ORs to combine conditions:
SELECT E_AUFMASS_KOMMENTARE.FIBU_FIRMA, E_AUFMASS_KOMMENTARE.AUFTR_NR, E_AUFMASS_KOMMENTARE.KOMMENTAR, AUFTR_EXT.ART_GRUPPE FROM HHNG_AU.E_AUFMASS_KOMMENTARE INNER JOIN HHNG_AU.AUFTR_EXT ON E_AUFMASS_KOMMENTARE.AUFTR_NR = AUFTR_EXT.AUFTR_NR WHERE (E_AUFMASS_KOMMENTARE.AUFTR_NR = '1248823' ) AND NOT EXISTS( SELECT * FROM HHNG_AU.EX_KOMMENTARE WHERE EX_KOMMENTARE.AUFTR_NR = '1248823' )
Can you please help me in updating DB2 table and is there a better way to update this huge table? Adv thxs.
UPDATE RT.ITEM IM SET
IM.ITEMNAME = GT.ITEM_D, IM.ITEMSIZE = GT.SIZE, IM.COLOR = GT.COL,
IM.ITEMINFO = GT.ITEM_I WHERE IM.RET = 14 AND IM.LAN = 10 and
IM.ITEMK IN ( SELECT GT.SN_N FROM GD.G_TEMP GT );
Trying to update a table(RT.ITEM) from another schema table(GD.G_TEMP) and getting below error msg:
[Code: -206, SQL State: 42703] DB2 SQL Error: SQLCODE=-206, SQLSTATE=42703, SQLERRMC=GT.ITEM_D
Your code won't work. DB2 doesn't support explicit JOIN in UPDATE. But you can do what you want with a correlated subquery:
UPDATE RT.ITEM IM
SET (ITEMNAME, ITEMSIZE, COLOR, ITEMINFO) =
(SELECT GT.ITEM_D, GT.SIZE, GT.COL, GT.ITEM_I
FROM GD.G_TEMP GT
WHERE GT.SN_N = IM.ITEMK
FETCH FIRST 1 ROW ONLY
)
WHERE IM.RET = 14 AND IM.LAN = 10 AND
EXISTS (SELECT 1
FROM GD.G_TEMP GT
WHERE GT.SN_N = IM.ITEMK
);
Hi you can try with merge command, if distinct don't solve your problems in gd_temp table with multiple rows for single sn_n value then you will have to add more filters in subquery.
MERGE INTO RT.ITEM IM
USING
(SELECT DISTINCT
GT.ITEM_D,
GT.SIZE,
GT.COL,
GT.ITEM_I
FROM GD.G_TEMP gt
) gt on gt.sn_n=im.itemk AND IM.RET = 14 AND IM.LAN = 10
WHEN MATCHED THEN UPDATE
SET (im.ITEMNAME, im.ITEMSIZE, im.COLOR, im.ITEMINFO) = (GT.ITEM_D, GT.SIZE, GT.COL, GT.ITEM_I)
I have one sql query like this :
select * from tbl_ServiceRequest SR left join tbl_Events e on s.SRId = e.SRId where CustomerId = 65 and convert(nvarchar(10),s.CreatedDate,120) > (select convert(nvarchar(10),OrganisationCycleDate,120) from tbl_OrganizationDetail where OrgId = (select OrgId from Organization))))
When i run it into the sql server 2008 than it runs perfact and gives the correct records.
But i am using PHP with zend framework and its giving this error in php :
[Microsoft][SQL Server Native Client 11.0][SQL Server]Incorrect syntax near ';'.
there is not any ';' in my query.
The problem is in the second condition. if i will remove the second condition than its not giving any error. Second condition is :
and convert(nvarchar(10),s.CreatedDate,120) > (select convert(nvarchar(10),OrganisationCycleDate,120) from tbl_OrganizationDetail where OrgId = (select OrgId from Organization))))
Its giving me this error when i try to download pdf of these records. I am using FPDF library for it. But it is error of SQL Server only.
Can anyone know how to solve this problem ?
Try this:
SELECT *
FROM tbl_ServiceRequest s
LEFT JOIN tbl_Events e ON s.SRId = e.SRId
WHERE CustomerId = 65
AND convert(nvarchar(10),s.CreatedDate,120) >
(SELECT convert(nvarchar(10),OrganisationCycleDate,120)
FROM tbl_OrganizationDetail
WHERE OrgId =
(SELECT OrgId
FROM Organization));
Renaming your tbl_ServiceRequest as s not SR I guess.
My Problem is solved. There isn't any problem with sql.
There is a problem with php.
I just replace ">" with > and "<" with < in my query and it runs perfactly.
Thanks a lot A.S. Roma for your help..