Query fails on tez but fine on MR.
We have tried many different permutations and combinations of set ... paarmeters but unable to run this query successfully in tez.
Query in MR runs in about 20 mins but tez engine, we saw that query was consuming huge memory in cluster, when this query runs, I see memory usage on cluster keep going up and reaches 100% on many nodes and query was failing with container exit on lost node message
tmp_track_tempo_category_link_rc is 600 Million rows and less than 10GB size on hdfs
set hive.execution.engine=tez;
DROP TEMPORARY FUNCTION row_rank;
CREATE TEMPORARY FUNCTION row_rank AS 'com.mycompany.hadoop.hive.udf.UDFRank';
SET hive.map.aggr=true;
SET hive.exec.parallel=true;
SET hive.merge.mapfiles=false;
--set hive.mapjoin.hybridgrace.hashtable=false ;
SET mapred.max.split.size=100000000;
SET mapred.reduce.tasks=32;
--set mapreduce.map.memory.mb=4096;
--set mapreduce.reduce.memory.mb=4096;
--set mapreduce.map.java.opts=-Xmx3686m;
--set mapreduce.reduce.java.opts=-Xmx3686m;
set hive.tez.container.size=6144;
set hive.tez.java.opts=-Xmx4096m ;
--set tez.am.task.max.failed.attempts=20 ;
--set tez.am.max.app.attempts=10;
--set tez.am.node-unhealthy-reschedule-tasks=true;
set tez.task.am.heartbeat.interval-ms.max=500 ;
--set tez.task.am.heartbeat.counter.interval-ms.max=8000;
--set tez.task.resource.memory.mb=2048;
set tez.am.resource.memory.mb=6144
set tez.am.java.opts=-Xmx4096m
set hive.vectorized.execution.enabled=true;
set hive.vectorized.execution.reduce.enabled=true;
set hive.exec.parallel=true;
set hive.auto.convert.join=true;
set hive.cbo.enable=true;
set hive.compute.query.using.stats=true;
set hive.stats.fetch.column.stats=true;
set hive.stats.fetch.partition.stats=true;
set hive.exec.compress.output=true;
set hive.exec.compress.intermediate=true;
--set hive.tez.container.size=1024;
--set hive.tez.java.opts=-Xmx512m;
set tez.runtime.io.sort.mb=2048;
set tez.grouping.min-size=16777216;
set tez.grouping.max-size=1073741824;
set tez.grouping.split-count=8;
set tez.am.resource.memory.mb=1024;
set hive.exec.reducers.bytes.per.reducer=10240000;
set hive.exec.reducers.max=100;
set hive.tez.auto.reducer.parallelism = true;
--set tez.runtime.unordered.output.buffer.size-mb=1024;
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.optimize.skewjoin=true ;
set hive.skewjoin.key=5000;
USE mydb;
SELECT track_id,
(category_id + 100000) category_id,
category_weight,
CAST (1 AS TINYINT) section_type_id
FROM (SELECT tmp4.*, row_rank (track_id) tempo_rank
FROM (SELECT tempo.track_id,
tempo.category_id,
tempo.category_weight
FROM tmp_track_tempo_category_link_rc tempo
JOIN
(SELECT tmp2.*
FROM (SELECT tmp1.track_id,
tmp1.apple_track_id,
row_rank (track_id) RANK
FROM ( SELECT b.track_id,
b.apple_track_id,
SUM (c.cnt) cnt,
SUM (
b.category_weight)
mx
FROM ( SELECT track_id,
category_id,
COUNT (
category_id)
cnt
FROM tmp_track_tempo_category_link_rc a
WHERE section_type_id =
1
GROUP BY track_id,
category_id)
c
JOIN
tmp_track_tempo_category_link_rc b
ON ( b.track_id =
c.track_id
AND b.category_id =
c.category_id
AND b.section_type_id =
1)
GROUP BY b.track_id,
b.apple_track_id
DISTRIBUTE BY track_id SORT BY track_id, cnt DESC, mx DESC, apple_track_id
) tmp1) tmp2
WHERE RANK = 1) tmp3
ON ( tmp3.track_id = tempo.track_id
AND tmp3.apple_track_id =
tempo.apple_track_id
AND tempo.section_type_id = 1)
DISTRIBUTE BY track_id SORT BY track_id, category_weight DESC, category_id
) tmp4) tmp5
WHERE tempo_rank <= 3 ;
Some of set now make memory situation better but it fails with Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : null
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:488)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:199)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : null
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.closeOp(MapJoinOperator.java:546)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.close(MapRecordProcessor.java:477)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator : null
Last I remember looking at Tez, it had a pretty simple-minded set of optimizations for sort and join exchanges. IT would be quite plausible if your (relatively large) query ran into those limits.
MapReduce doesn't have these problems because every step is completely written to disk so there aren't any in-memory joins or sorts. That leads to really slow performance sometimes, but it also gives some simple-minded guarantees and robustness.
Have you asked on the appropriate Apache mailing list? I know that Cloudera has pretty much abandoned the Hadoop community, but there are likely still some fairly astute people around.
Related
If an object (table/indexes..) is dropped, will the statistics related to that object be still available or gets dropped along with the object that is dropped?
From virtual-dba.com
Just Note when Dropping a Table/Indexes: If any statistic types are
associated with the table, then the database disassociates the
statistics types and removes any user-defined statistics collected
with the statistics type.
What is default trace?
The default trace is a new feature first included with SQL Server 2005 which provides auditing on schema modifications such as table creation and stored procedure deletion and the like. It does run by default, but you can turn it on ..
For more info: https://www.mssqltips.com/sqlservertip/1739/using-the-default-trace-in-sql-server/
In this query.. use EventClass 92 and 93 to track database auto-growth events. Here is the query to find who dropped / created or altered object in database or database itself.
DECLARE #current VARCHAR(500);
DECLARE #start VARCHAR(500);
DECLARE #indx INT;
SELECT #current = path
FROM sys.traces
WHERE is_default = 1;
SET #current = REVERSE(#current)
SELECT #indx = PATINDEX('%\%', #current)
SET #current = REVERSE(#current)
SET #start = LEFT(#current, LEN(#current) - #indx) + '\log.trc';
-- Change filter as needed
SELECT
CASE EventClass
WHEN 46 THEN 'Object:Created'
WHEN 47 THEN 'Object:Deleted'
WHEN 164 THEN 'Object:Altered'
END,
DatabaseName, ObjectName, HostName, ApplicationName, LoginName,
StartTime
FROM
::fn_trace_gettable(#start, DEFAULT)
WHERE
EventClass IN (46,47,164) AND EventSubclass = 0 AND DatabaseID <> 2
ORDER BY
StartTime DESC
You can use this query for finding all details of created, deleted, alter table info...
Here some different default trace in SQL Server....
SELECT DISTINCT Trace.EventID, TraceEvents.NAME AS Event_Desc
FROM ::fn_trace_geteventinfo(1) Trace,
sys.trace_events TraceEvents
WHERE Trace.eventID = TraceEvents.trace_event_id
This above query is used for Event Class and also in above example using (Object:Created, Object:Deleted, Object:Altered); you can change as per your requirement.
My script keeps falling with the same error
java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support
The code itself looks like this
WITH step1 AS(
SELECT columns
FROM t1 stg
WHERE time_key < '2017-04-08' AND time_key >= DATE_ADD('2017-04-08', -31)
GROUP BY columns
HAVING conditions1
)
, step2 AS(
SELECT columns
FROM t2
WHERE conditions2
)
, step3 AS(
SELECT columns
FROM stg
JOIN comverse_sub
ON conditions3
)
INSERT INTO TABLE t1 PARTITION(time_key = '2017-04-08')
SELECT columns
FROM step3
WHERE conditions4
I checked if there is snappy installed
hadoop checknative -a
and got
snappy: true /usr/hdp/2.5.0.0-1245/hadoop/lib/native/libsnappy.so.1
My settings for tez are
set tez.queue.name=adhoc;
set hive.execution.engine=tez;
set hive.tez.container.size=4096;
set hive.auto.convert.join=true;
set hive.exec.parallel=true;
set hive.tez.auto.reducer.parallelism=true;
SET hive.exec.compress.output=true;
SET tez.runtime.compress=true;
SET tez.runtime.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
I should also note that not all script on tez fail. Some work. Like this one
WITH hist AS(
SELECT columns
FROM t1
WHERE conditions1
)
INSERT INTO TABLE t1 PARTITION(time_key)
SELECT columns
FROM hist
INNER JOIN t2
on conditions2
INNER JOIN t3
ON conditions3
WHERE conditions4
Why does this happen?
I checked this and this and this. Didn't help. Also, when I run the scripts on MR they all work.
Well, I solved it. I just kept adding settings untill it worked. Still don't know what the problem was and the solution is not very good, but it'll do for the time being.
set tez.queue.name=adhoc;
set hive.execution.engine=tez;
set hive.tez.container.size=4096;
set hive.auto.convert.join=true;
set hive.exec.parallel=true;
set hive.tez.auto.reducer.parallelism=true;
SET hive.exec.compress.output=true;
SET tez.runtime.compress=true;
SET tez.runtime.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET das.reduce-tasks-per-node=12;
SET das.map-tasks-per-node=12;
SET das.job.map-task.memory=4096;
SET das.job.reduce-task.memory=4096;
SET das.job.application-manager.memory=4096;
SET tez.runtime.io.sort.mb=512;
SET tez.runtime.io.sort.factor=100;
set hive.tez.min.partition.factor=0.25;
set hive.tez.max.partition.factor=2.0;
set mapred.reduce.tasks = -1;
My stored procedure is like this
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[Bid_Create]
#BidType int,
#ClientId int,
#BidDate date,
#EmailNotificationStatus int,
#BidStatus int,
#BidAmount int,
#ProductId int
AS
DECLARE #highestBid int;
BEGIN
BEGIN TRY
BEGIN TRANSACTION
SET NOCOUNT ON;
SET #highestBid = (SELECT Max(wf_bid.BidAmount) AS HighestBitAmount
FROM wf_bid
WHERE wf_bid.ProductId = #ProductId)
IF #highestBid is NULL OR #highestBid < #BidAmount
BEGIN
UPDATE wf_bid
SET BidStatus = '1'
WHERE Id = (SELECT TOP 1 id
FROM [wf_bid]
WHERE BidAmount = (SELECT MAX(BidAmount)
FROM [wf_bid]
WHERE ProductId = #ProductId
AND ClientId = #ClientId))
INSERT INTO wf_bid (BidType, ClientId, BidDate, EmailNotificationStatus, BidStatus)
VALUES (#BidType, #ClientId, #BidDate, #EmailNotificationStatus, #BidStatus)
END
COMMIT TRANSACTION
END TRY
BEGIN CATCH
ROLLBACK TRANSACTION
END CATCH
END
Everything looks okay to me. But once I run this, table is getting locked. No other query on the table works (I think it is because transaction is not getting committed).
Can anyone point out what is wrong with this query? And how can I unlock the table?
But once I run this, table is getting locked
This may be due to update taking many locks,which in turn may be due to predicate not being sargable. Though this update locks(U) lock will be released as soon as the predicate is not matched.You will experience blocking
One more reason ,why this update may block your whole table is when this transaction acquires more than 5000 locks..
another reason can be when your transaction fails after committing so many rows and it has to do a lot of rollback work
Above are the reasons ,i could think of,where you can experience table is locked feeling
to troubleshoot that,you will need to check lockings blokcings using below query
select resource_type,resource_Database_id,
request_mode,request_type,request_Status,request_session_id
from sys.dm_tran_locks
where request_session_id=<<your update session id>>
also you are accessing table many times,for getting max.you can rewrite it like below
;with cte
as
(
select top (1) with ties id,bidstatus from
wf_bid
where ProductId=#ProductId and ClientId=#ClientId)
order by
row_number() over (partition by id order by bid_Amount desc)
)
update cte
set bidstatus=1
I am trying to insert 1,500,000 records into a table. Am facing table lock issues during the insertion. So I came up with the below batch insert.
DECLARE #BatchSize INT = 50000
WHILE 1 = 1
BEGIN
INSERT INTO [dbo].[Destination]
(proj_details_sid,
period_sid,
sales,
units)
SELECT TOP(#BatchSize) s.proj_details_sid,
s.period_sid,
s.sales,
s.units
FROM [dbo].[SOURCE] s
WHERE NOT EXISTS (SELECT 1
FROM dbo.Destination d
WHERE d.proj_details_sid = s.proj_details_sid
AND d.period_sid = s.period_sid)
IF ##ROWCOUNT < #BatchSize
BREAK
END
I have a clustered Index on Destination table (proj_details_sid ,period_sid ). NOT EXISTS part is just to restrict inserted records from again inserting into the table
Am I doing it right, will this avoid table lock ? or is there any better way.
Note : Time taken is more or less same with batch and without batch insert
Lock escalation is not likely to be related to the SELECT part of your statement at all.
It is a natural consequence of inserting a large number of rows
Lock escalation is triggered when lock escalation is not disabled on the table by using the ALTER TABLE SET LOCK_ESCALATION option, and when either of the following conditions exists:
A single Transact-SQL statement acquires at least 5,000 locks on a single nonpartitioned table or index.
A single Transact-SQL statement acquires at least 5,000 locks on a single partition of a partitioned table and the ALTER TABLE SET LOCK_ESCALATION option is set to AUTO.
The number of locks in an instance of the Database Engine exceeds memory or configuration thresholds.
If locks cannot be escalated because of lock conflicts, the Database Engine periodically triggers lock escalation at every 1,250 new locks acquired.
You can easily see this for yourself by tracing the lock escalation event in Profiler or simply trying the below with different batch sizes. For me TOP (6228) shows 6250 locks held but TOP (6229) it suddenly plummets to 1 as lock escalation kicks in. The exact numbers may vary (dependant on database settings and resources currently available). Use trial and error to find the threshold where lock escalation appears for you.
CREATE TABLE [dbo].[Destination]
(
proj_details_sid INT,
period_sid INT,
sales INT,
units INT
)
BEGIN TRAN --So locks are held for us to count in the next statement
INSERT INTO [dbo].[Destination]
SELECT TOP (6229) 1,
1,
1,
1
FROM master..spt_values v1,
master..spt_values v2
SELECT COUNT(*)
FROM sys.dm_tran_locks
WHERE request_session_id = ##SPID;
COMMIT
DROP TABLE [dbo].[Destination]
You are inserting 50,000 rows so almost certainly lock escalation will be attempted.
The article How to resolve blocking problems that are caused by lock escalation in SQL Server is quite old but a lot of the suggestions are still valid.
Break up large batch operations into several smaller operations (i.e. use a smaller batch size)
Lock escalation cannot occur if a different SPID is currently holding an incompatible table lock - The example they give is a different session executing
BEGIN TRAN
SELECT * FROM mytable (UPDLOCK, HOLDLOCK) WHERE 1=0
WAITFOR DELAY '1:00:00'
COMMIT TRAN
Disable lock escalation by enabling trace flag 1211 - However this is a global setting and can cause severe issues. There is a newer option 1224 that is less problematic but this is still global.
Another option would be to ALTER TABLE blah SET (LOCK_ESCALATION = DISABLE) but this is still not very targeted as it affects all queries against the table not just your single scenario here.
So I would opt for option 1 or possibly option 2 and discount the others.
Instead of checking the data exists in Destination, it seems better to store all data in temp table first, and batch insert into Destination
Reference: Using ROWLOCK in an INSERT statement (SQL Server)
DECLARE #batch int = 100
DECLARE #curRecord int = 1
DECLARE #maxRecord int
-- remove (nolock) if you don't want to have dirty read
SELECT row_number over (order by s.proj_details_sid, s.period_sid) as rownum,
s.proj_details_sid,
s.period_sid,
s.sales,
s.units
INTO #Temp
FROM [dbo].[SOURCE] s WITH (NOLOCK)
WHERE NOT EXISTS (SELECT 1
FROM dbo.Destination d WITH (NOLOCK)
WHERE d.proj_details_sid = s.proj_details_sid
AND d.period_sid = s.period_sid)
-- change this maxRecord if you want to limit the records to insert
SELECT #maxRecord = count(1) from #Temp
WHILE #maxRecord >= #curRecord
BEGIN
INSERT INTO [dbo].[Destination]
(proj_details_sid,
period_sid,
sales,
units)
SELECT proj_details_sid, period_sid, sales, units
FROM #Temp
WHERE rownum >= #curRecord and rownum < #curRecord + #batch
SET #curRecord = #curRecord + #batch
END
DROP TABLE #Temp
I added (NOLOCK) your destination table -> dbo.Destination(NOLOCK).
Now, You won't lock your table.
WHILE 1 = 1
BEGIN
INSERT INTO [dbo].[Destination]
(proj_details_sid,
period_sid,
sales,
units)
SELECT TOP(#BatchSize) s.proj_details_sid,
s.period_sid,
s.sales,
s.units
FROM [dbo].[SOURCE] s
WHERE NOT EXISTS (SELECT 1
FROM dbo.Destination(NOLOCK) d
WHERE d.proj_details_sid = s.proj_details_sid
AND d.period_sid = s.period_sid)
IF ##ROWCOUNT < #BatchSize
BREAK
END
To do this you can use WITH (NOLOCK) in your select statement.
BUT NOLOCK is not recommended on OLTP Databases.
I'm currently merging large data sets in Windows Azure SQL Database. I was wondering if there was a way not to get the following error:
40552 : The session has been terminated because of excessive transaction log space usage. Try modifying fewer rows in a single transaction.
My data sets are larger than 15 million records.
From an application find the number of record that need to be imported and based on the batch size calculate the number of batches that need to be executed. Then for each batch, create a new SQL Command which executes the merge Stored Procedure with the right offset Index. Be sure to insert a short delay between each SQL Command, because if they run too fast Windows Azure SQL Database might raise a 40501 due to an excessive number of requests.
CREATE PROCEDURE [dbo].[MergeCustomerTables]
#offsetIndex int = 0,
#batchSize int = 10000
AS
DECLARE #offset as bigint
SELECT #offset = #offsetIndex * #batchSize
MERGE Customers AS Target
USING (SELECT a.[Name], a.[LastName], a.[Email] from CustomerInsertTable as a
ORDER BY a.[Email]
OFFSET #offset ROWS
FETCH NEXT #batchSize ROWS ONLY
) AS Source
ON (Target.[Email] = Source.[Email])
WHEN MATCHED THEN
UPDATE SET Target.[Name] = Source.[Name]
, Target.[LastName] = Source.[LastName]
WHEN NOT MATCHED BY TARGET THEN
INSERT ([Name], [LastName], [Email])
VALUES (
Source.[Name]
, Source.[LastName]
, Source.[Email]
);