I've been running into an issue where select top 1 * from x.y will sit in queue for up to an hour whereas a select * from x.y sample 1 will run within seconds. I've been told that it's because select top is collecting all the data, sorting it and grabbing the top 1 and sample 1 is just grabbing a random row, no sorting the data at all, just a random row.
This explanation doesn't makes sense to me especially since I'm not including an order by in the query and the database usage metrics for sample are orders of magnitude worse.
AMPCPUTime
StartTime
FirstStepTime
FirstRespTime
Spool_Usage_GB
TotalIOCount
QueryText
2.92
10:57:37.52
10:57:37.52
10:57:37.92
0
24248
select * from x.y sample 1;
0.02
9:30:08.39
10:57:37.10
10:57:37.26
0
55
SELECT TOP 1 * FROM x.y;
EXPLAIN SELECT TOP 1 * FROM x.y;
First, we lock x.y in TD_MAP1 for read
on a reserved RowHash to prevent global deadlock.
Next, we lock x.y in TD_MAP1 for read.
We do an all-AMPs STAT FUNCTION step in TD_MAP1 from
x.y by way of an all-rows scan with no
residual conditions into Spool 5, which is built locally on the
AMPs in TD_Map1. The result rows are put into Spool 1
(group_amps), which is built locally on the AMPs. This step is
used to retrieve the TOP 1 rows. One AMP is randomly selected to
retrieve 1 rows. If this step retrieves less than 1 rows, then
execute step 4. The size is estimated with high confidence to be
1 row (594 bytes). The estimated time for this step is 1 minute
and 34 seconds.
We do an all-AMPs STAT FUNCTION step in TD_MAP1 from
x.y by way of an all-rows scan with no
residual conditions into Spool 5 (Last Use), which is
redistributed by hash code to all AMPs in TD_Map1. The result
rows are put into Spool 1 (group_amps), which is built locally on
the AMPs. This step is used to retrieve the TOP 1 rows. The size
is estimated with high confidence to be 1 row (594 bytes). The
estimated time for this step is 1 minute and 34 seconds.
We do a group-AMP SORT to order Spool 1 (group_amps) by the sort
key in spool field1.
Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> The contents of Spool 1 are sent back to the user as the result of
statement 1.
EXPLAIN select * from x.y sample 1;
First, we lock x.y in TD_MAP1 for read
on a reserved RowHash to prevent global deadlock.
Next, we lock x.y in TD_MAP1 for read.
We do an all-AMPs SAMPLING step in TD_MAP1 from
x.y by way of an all-rows scan with no
residual conditions into Spool 1 (group_amps), which is built
locally on the AMPs. Samples are specified as a number of rows.
The size of Spool 1 is estimated with high confidence to be 1 row
(594 bytes).
Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> The contents of Spool 1 are sent back to the user as the result of
statement 1.
Related thread: Efficiency of top vs. sample in Teradata
Related
I am using Teradata database but I am quite new to its functioning. Could you please help me with making the below query more efficient so that it does not yield 'no more spool space' error? It is getting too heavy after I add the 2nd join.
SELECT
a.src_cmpgn_code,
a.cmc_name,
SUM(b.open_cnt)
FROM access_views.dw_cmc_lkp a
LEFT JOIN prs_restricted_v.mh_crm_engmnt_sd b
ON b.cmpgn_id = a.cmc_id
LEFT JOIN access_views.dw_cmc_instnc c
ON b.cmpgn_id = c.cmc_id
WHERE 1=1
AND b.trigger_dt BETWEEN '2019-01-01' AND '2019-12-31'
AND b.site_cntry_id = 1
AND a.cmpgn_group_name IN ('a', 'b', 'c', 'd')
AND c.dlvry_vhcl_id IN (1, 10)
AND c.chnl_id = 1
GROUP BY 1,2;
Explain looks like this:
This query is optimized using type 2 profile Cost_NoSlidingJ_Profile,
profileid 10007. 1) First, we lock mdm_tables.DW_CMC_INSTNC in view
access_views.dw_cmc_instnc for access, we lock MDM_TABLES.DW_CMC_LKP
in view access_views.dw_cmc_lkp for access, and we lock
PRS_T.MH_CRM_ENGMNT_SD in view prs_restricted_v.mh_crm_engmnt_sd for
access. 2) Next, we do an all-AMPs RETRIEVE step from 365 partitions
of PRS_T.MH_CRM_ENGMNT_SD in view prs_restricted_v.mh_crm_engmnt_sd
with a condition of ("(NOT (PRS_T.MH_CRM_ENGMNT_SD in view
prs_restricted_v.mh_crm_engmnt_sd.CMPGN_ID IS NULL )) AND
(((PRS_T.MH_CRM_ENGMNT_SD in view
prs_restricted_v.mh_crm_engmnt_sd.TRIGGER_DT <= DATE '2019-12-31') AND
(PRS_T.MH_CRM_ENGMNT_SD.TRIGGER_DT >= DATE '2019-01-01')) AND
(PRS_T.MH_CRM_ENGMNT_SD in view
prs_restricted_v.mh_crm_engmnt_sd.SITE_CNTRY_ID = 1. ))") into Spool 4
(all_amps), which is redistributed by the hash code of (
PRS_T.MH_CRM_ENGMNT_SD.CMPGN_ID) to all AMPs. The size of Spool 4 is
estimated with no confidence to be 329,656,959 rows ( 7,582,110,057
bytes). The estimated time for this step is 2.40 seconds. 3) We do an
all-AMPs JOIN step from MDM_TABLES.DW_CMC_LKP in view
access_views.dw_cmc_lkp by way of an all-rows scan with a condition of
("MDM_TABLES.DW_CMC_LKP in view
access_views.dw_cmc_lkp.CMPGN_GROUP_NAME IN ('Bucks_Nectar_eBayPlus',
'DailyDeal','Other','STEP_User_Agreement')"), which is joined to Spool
4 (Last Use) by way of an all-rows scan. MDM_TABLES.DW_CMC_LKP and
Spool 4 are joined using a single partition hash join, with a join
condition of ("CMPGN_ID = MDM_TABLES.DW_CMC_LKP.CMC_ID"). The result
goes into Spool 5 (all_amps) fanned out into 5 hash join partitions,
which is built locally on the AMPs. The size of Spool 5 is estimated
with no confidence to be 79,119,821 rows (10,681,175,835 bytes). The
estimated time for this step is 0.19 seconds. 4) We do an all-AMPs
RETRIEVE step from mdm_tables.DW_CMC_INSTNC in view
access_views.dw_cmc_instnc by way of an all-rows scan with a condition
of ("(mdm_tables.DW_CMC_INSTNC in view
access_views.dw_cmc_instnc.DLVRY_VHCL_ID IN (1 , 10 )) AND
((mdm_tables.DW_CMC_INSTNC in view access_views.dw_cmc_instnc.CHNL_ID
= 1) AND (mdm_tables.DW_CMC_INSTNC in view access_views.dw_cmc_instnc.TRTMNT_TYPE_CODE <> 'I'))") into Spool 6
(all_amps) fanned out into 5 hash join partitions, which is
redistributed by the hash code of ( mdm_tables.DW_CMC_INSTNC.CMC_ID)
to all AMPs. The size of Spool 6 is estimated with no confidence to be
2,874,675 rows (48,869,475 bytes). The estimated time for this step is
0.58 seconds. 5) We do an all-AMPs JOIN step from Spool 5 (Last Use) by way of an all-rows scan, which is joined to Spool 6 (Last Use) by
way of an all-rows scan. Spool 5 and Spool 6 are joined using a hash
join of 5 partitions, with a join condition of ("(CMPGN_ID = CMC_ID)
AND (CMC_ID = CMC_ID)"). The result goes into Spool 3 (all_amps),
which is built locally on the AMPs. The size of Spool 3 is estimated
with no confidence to be 5,353,507,625 rows ( 690,602,483,625 bytes).
The estimated time for this step is 14.82 seconds. 6) We do an
all-AMPs SUM step to aggregate from Spool 3 (Last Use) by way of an
all-rows scan , grouping by field1 (
MDM_TABLES.DW_CMC_LKP.SRC_CMPGN_CODE ,MDM_TABLES.DW_CMC_LKP.CMC_NAME).
Aggregate Intermediate Results are computed globally, then placed in
Spool 7. The size of Spool 7 is estimated with no confidence to be
11,774 rows (5,286,526 bytes). The estimated time for this step is
24.51 seconds. 7) We do an all-AMPs RETRIEVE step from Spool 7 (Last Use) by way of an all-rows scan into Spool 1 (group_amps), which is
built locally on the AMPs. The size of Spool 1 is estimated with no
confidence to be 11,774 rows (2,837,534 bytes). The estimated time for
this step is 0.01 seconds. 8) Finally, we send out an END TRANSACTION
step to all AMPs involved in processing the request. -> The contents
of Spool 1 are sent back to the user as the result of statement 1. The
total estimated time is 42.50 seconds.
I want to insert data into the target table but unfortunately it takes too much time, even it is only around 800 000 records.
I think that the problem is with the execution plan/bad indexes or statistics, however, can you please look at the explain plan whether something is suspected there?
Here is the plan:
This query is optimized using type 2 profile nonested_cost, profileid
10003.
This request is eligible for incremental planning and execution (IPE)
but does not meet cost thresholds. The following is the static plan
for the request.
1) First, we lock Schema1.Target_Table in TD_MAP1
for write on a reserved RowHash to prevent global deadlock.
2) Next, we lock Schema1.Target_Table in TD_MAP1
for write, we lock Schema1.Table1 in view
Schema1.Table1_View in TD_MAP1 for access, and we lock
Schema1.Table2 in view
Schema1.Table3 in TD_MAP1 for access.
3) We do an all-AMPs RETRIEVE step in TD_MAP1 from
Schema1.Table2 in view
Schema1.Table3 by way of an all-rows scan
with a condition of ("(NOT (Schema1.Table2 in
view Schema1.Table3.CUSTOMER_ID IS NULL
)) AND (Schema1.Table2 in view
Schema1.Table3.TYPE = 2.)") into Spool 3
(all_amps) (compressed columns allowed), which is redistributed by
the hash code of (
Schema1.Table2.CUSTOMER_ID) to all AMPs in
TD_Map1. The size of Spool 3 is estimated with high confidence to
be 66,211 rows (66,939,321 bytes). The estimated time for this
step is 0.80 seconds.
4) We do an all-AMPs JOIN step in TD_Map1 from Spool 3 (Last Use) by
way of a RowHash match scan, which is joined to
Schema1.Table1 in view Schema1.Table1_View by way
of a RowHash match scan with no residual conditions. Spool 3 and
Schema1.Table1 are joined using a single partition hash
join, with a join condition of ("CUSTOMER_ID =
Schema1.Table1.CUSTOMER_ID"). The result goes into
Spool 4 (all_amps) (compressed columns allowed), which is
redistributed by the hash code of (
HERE IS THE LIST OF COLUMNS FROM TABLE 1 AND TABLE 2 to all AMPs in TD_Map1.
Then we do a SORT to order Spool 4 by row hash. The size of Spool
4 is estimated with index join confidence to be 66,211 rows (
73,163,155 bytes). The estimated time for this step is 0.11
seconds.
5) We execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step in TD_Map1 from Spool 4 by
way of an all-rows scan into Spool 5 (all_amps) (compressed
columns allowed) fanned out into 33 hash join partitions,
which is duplicated on all AMPs in TD_Map1. The size of
Spool 5 is estimated with index join confidence to be
28,603,152 rows (31,635,086,112 bytes). The estimated time
for this step is 1.69 seconds.
2) We do an all-AMPs RETRIEVE step in TD_MAP1 from
S.TAB_M by way of an all-rows scan with a
condition of ("(NOT (S.TAB_M.EQUIPMENT_ID IS NULL ))
AND (NOT (S.TAB_M.COUNTY_ID IS NULL ))")
into Spool 6 (all_amps) (compressed columns allowed) fanned
out into 33 hash join partitions, which is built locally on
the AMPs. The size of Spool 6 is estimated with high
confidence to be 120,648,009 rows (29,679,410,214 bytes).
The estimated time for this step is 2.68 seconds.
6) We do an all-AMPs JOIN step in TD_Map1 from Spool 6 (Last Use) by
way of an all-rows scan, which is joined to Spool 5 (Last Use) by
way of an all-rows scan. Spool 6 and Spool 5 are joined using a
hash join of 33 partitions, with a join condition of ("(NOT
(SubCustomer_ID IS NULL )) AND ((NOT (ORG_ID IS NULL )) AND
((EQUIPMENT_ID = SubCustomer_ID) AND ((COUNTY_ID )= (ORG_ID
(FLOAT, FORMAT '-9.99999999999999E-999')))))"). The result goes
into Spool 7 (all_amps) (compressed columns allowed), which is
redistributed by the hash code of (
HERE IS THE LIST OF COLUMNS FROM TABLE 1 AND TABLE 2) to all AMPs in
TD_Map1. Then we do a SORT to order Spool 7 by row hash. The
size of Spool 7 is estimated with index join confidence to be
88,525 rows (28,239,475 bytes).
7) We do an all-AMPs JOIN step in TD_Map1 from Spool 7 (Last Use) by
way of a RowHash match scan, which is joined to Spool 4 (Last Use)
by way of a RowHash match scan. Spool 7 and Spool 4 are
right outer joined using a merge join, with a join condition of (
"Field_1 = Field_1"). The result goes into Spool 2 (all_amps)
(compressed columns allowed), which is built locally on the AMPs.
The size of Spool 2 is estimated with index join confidence to be
88,525 rows (114,639,875 bytes). The estimated time for this step
is 1.68 seconds.
8) We do an all-AMPs SUM step in TD_Map1 to aggregate from Spool 2
(Last Use) by way of an all-rows scan, grouping by field1 (HERE IS THE LIST OF COLUMNS FROM TABLE 1 AND TABLE 2 AND S.TAB_M). Aggregate Intermediate
Results are computed locally, then placed in Spool 11 in TD_Map1.
The size of Spool 11 is estimated with low confidence to be 66,394
rows (330,575,726 bytes). The estimated time for this step is
0.23 seconds.
9) We do an all-AMPs RETRIEVE step in TD_Map1 from Spool 11 (Last
Use) by way of an all-rows scan into Spool 1 (all_amps)
(compressed columns allowed), which is redistributed by the hash
code of ((CASE WHEN (NOT (S.TAB_M.TIME_DELIVERY IS
NULL )) THEN (S.TAB_M.TIME_DELIVERY) WHEN (NOT
(S.TAB_M.TIME_DELIVERY_EXP IS NULL )) THEN
(S.TAB_M.TIME_DELIVERY_EXP) ELSE (0) END )(INTEGER)) to
all AMPs in TD_Map1. Then we do a SORT to order Spool 1 by row
hash. The size of Spool 1 is estimated with low confidence to be
66,394 rows (85,050,714 bytes). The estimated time for this step
is 0.05 seconds.
10) We do an all-AMPs MERGE step in TD_MAP1 into
Schema1.Target_Table from Spool 1 (Last Use).
The size is estimated with low confidence to be 66,394 rows. The
estimated time for this step is 4.00 seconds.
11) We spoil the parser's dictionary cache for the table.
12) Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> No rows are returned to the user as the result of statement 1.```
I am working in Teradata on a legacy SQL query used to make a temp table that pushes data to a downstream data analytics application. I am new to the job, and after taking a quick look at the table, I noticed an error where the query was referencing the same table within the join conditions. After fixing it, I re-ran the query, expecting better results.
The database holds a couple billion records. Before fixing the query, the original temp table contained around 2.5 million rows, and took about 5 seconds to run. After fixing the query, the query will run for hours with no signs of progress, then usually crashes the application.
The original query is below. I put a comment in where I 'fixed' the join.
I ran just the select statement, and used sample 10 just to see if I can pull any data back, but again, it's taking forever to run.
Any thoughts on why the query isn't executing? Or how to improve performance to see if it's actually working? Or even how to check if anything is happening under the hood?
create multiset volatile table order_date as
(
SELECT a.ord_nbr,
a.ord_spot_cnt AS ord_spot_cnt,
a.ord_totl_amt AS ord_totl_amt,
a.eclipse_regn_nm AS eclipse_regn_nm,
a.sale_offc_key AS sales_office,
so.sale_offc_nm AS sales_office_name,
a.sale_prsn_key AS sales_person,
sp.sale_prsn_nm AS sales_person_name,
trunc(a.ctrc_begn_dt, 'month') ctrc_mnth,
(cast(a.ctrc_end_dt AS date format 'yyyy/mm/dd') - cast(a.ctrc_begn_dt AS date format 'yyyy/mm/dd')) AS ctrc_duration,
CASE WHEN extract (month FROM a.ctrc_begn_dt) in (10,11) THEN 1
WHEN extract (month FROM a.ctrc_begn_dt) in (12) AND extract (day FROM a.ctrc_begn_dt) <= 22 THEN 1 ELSE 0 END AS first_look_flg,
CASE WHEN extract (month FROM a.ctrc_begn_dt) in (12) AND extract (day FROM a.ctrc_begn_dt) > 22 THEN 2
WHEN extract (month FROM a.ctrc_begn_dt) in (1) THEN 2
WHEN extract (month FROM a.ctrc_begn_dt) in (2) AND extract (day FROM a.ctrc_begn_dt) <= 14 THEN 1 ELSE 0 END AS second_look_flg
FROM PRD_AM_BI_1.am_order_eda_fact a
inner join prd_am_bi_1.AM_SALES_OFFICE_EDA_DIM so
on (a.SALE_OFFC_KEY = so.SALE_OFFC_KEY
and so.ECLIPSE_REGN_NM = so.ECLIPSE_REGN_NM)
-- I 'fixed' the above line to "and a.ECLIPSE_REGN_NM = so.ECLIPSE_REGN_NM)"
inner join prd_am_bi_1.AM_SALES_PERSON_EDA_DIM sp
on (a.SALE_PRSN_KEY = sp.SALE_PRSN_KEY
and a.ECLIPSE_REGN_NM = sp.ECLIPSE_REGN_NM)
)
with data
on commit preserve rows
;
EDIT:
Here is the EXPLAIN output for the 'fixed query. Both queries have the same total estimated time, so I'm not sure why the 'fixed' one is not actually executing.
Explanation
------------------------------------------------------------------------
1) First, we lock PRD_AM_1.AM_ORDER_EDA_FACT in view
PRD_AM_BI_1.am_order_eda_fact in TD_MAP1 for access, we lock
PRD_AM_1.AM_SALES_PERSON_EDA_DIM in view
PRD_AM_BI_1.AM_SALES_PERSON_EDA_DIM in TD_MAP1 for access, and we
lock PRD_AM_1.AM_SALES_OFFICE_EDA_DIM in view
PRD_AM_BI_1.AM_SALES_OFFICE_EDA_DIM in TD_MAP1 for access.
2) Next, we execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step in TD_MAP1 from
PRD_AM_1.AM_SALES_OFFICE_EDA_DIM in view
PRD_AM_BI_1.AM_SALES_OFFICE_EDA_DIM by way of an all-rows
scan with no residual conditions into Spool 2 (all_amps),
which is redistributed by hash code to all AMPs to all AMPs
in TD_Map1. Then we do a SORT to order Spool 2 by row hash.
The size of Spool 2 is estimated with high confidence to be
426 rows (21,300 bytes). The estimated time for this step is
0.01 seconds.
2) We do an all-AMPs RETRIEVE step in TD_MAP1 from
PRD_AM_1.AM_SALES_PERSON_EDA_DIM in view
PRD_AM_BI_1.AM_SALES_PERSON_EDA_DIM by way of an all-rows
scan with no residual conditions into Spool 3 (all_amps),
which is redistributed by hash code to all AMPs to all AMPs
in TD_Map1. Then we do a SORT to order Spool 3 by row hash.
The size of Spool 3 is estimated with high confidence to be
229,155 rows (10,770,285 bytes). The estimated time for this
step is 0.04 seconds.
3) We do an all-AMPs RETRIEVE step in TD_MAP1 from
PRD_AM_1.AM_ORDER_EDA_FACT in view
PRD_AM_BI_1.am_order_eda_fact by way of an all-rows scan with
no residual conditions into Spool 4 (all_amps), which is
redistributed by hash code to all AMPs to all AMPs in TD_Map1.
Then we do a SORT to order Spool 4 by row hash. The size of
Spool 4 is estimated with high confidence to be 2,041,151
rows (136,757,117 bytes). The estimated time for this step
is 0.21 seconds.
3) We do an all-AMPs JOIN step in TD_Map1 from Spool 2 (Last Use) by
way of a RowHash match scan, which is joined to Spool 3 (Last Use)
by way of a RowHash match scan. Spool 2 and Spool 3 are joined
using a merge join, with a join condition of ("ECLIPSE_REGN_NM =
ECLIPSE_REGN_NM"). The result goes into Spool 5 (all_amps), which
is built locally on the AMPs. The size of Spool 5 is estimated
with no confidence to be 203,927 rows (17,129,868 bytes). The
estimated time for this step is 0.02 seconds.
4) We do an all-AMPs JOIN step in TD_Map1 from Spool 4 (Last Use) by
way of a RowHash match scan, which is joined to Spool 5 (Last Use)
by way of a RowHash match scan. Spool 4 and Spool 5 are joined
using a merge join, with a join condition of ("(ECLIPSE_REGN_NM =
ECLIPSE_REGN_NM) AND ((SALE_PRSN_KEY = SALE_PRSN_KEY) AND
((ECLIPSE_REGN_NM = ECLIPSE_REGN_NM) AND (SALE_OFFC_KEY =
SALE_OFFC_KEY )))"). The result goes into Spool 1 (group_amps),
which is built locally on the AMPs. The size of Spool 1 is
estimated with no confidence to be 2,041,151 rows (293,925,744
bytes). The estimated time for this step is 0.02 seconds.
-> The contents of Spool 1 are sent back to the user as the result of
statement 1. The total estimated time is 0.24 seconds.
Review the explain plan to see where the query is spending the most cost. Then focus on reducing that cost by using different join, filter, subquery, or making use of an index.
There is a snipped of product code that does some row check. It's actually migrated code that came into teradata and no one has bothered to change it to be TD savvy, should I say.
This code now throws
2646 : No More spool...
Error and that is not really a spool shortage but due to data-skew as would be evident to any Teradata Master.
Code logic is plain stupid but they are running it in Prod. Code change is NOT an option now because this is production. I can rewrite it using a Simple NOT Exists and the Query will run fine.
EXPLAIN SELECT ((COALESCE(FF.SKEW_COL,-99999))) AS Cnt1,
COUNT(*) AS Cnt
FROM DB.10_BILLON_FACT FF
WHERE FF.SKEW_COL IN(
SELECT F.SKEW_COL
FROM DB.10_BILLON_FACT F
EXCEPT
SELECT D.DIM_COL
FROM DB.Smaller_DIM D
)
Its failing because it wants to redistribute on SKEW_COL. WHATEVER I DO THIS WILL NOT CHANGE. SKEW_COL is 99% skewed.
here's the explain.FAILS ON STEP # 4.1
This query is optimized using type 2 profile insert-sel, profileid
10001.
1) First, we lock a distinct DB."pseudo table" for read on a
RowHash to prevent global deadlock for DB.F.
2) Next, we lock a distinct DB."pseudo table" for read on a
RowHash to prevent global deadlock for DB.D.
3) We lock DB.F for read, and we lock DB.D for read.
4) We execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step from DB.F by way of an
all-rows scan with no residual conditions into Spool 6
(all_amps), which is redistributed by the hash code of (
DB.F.SKEW_COL) to all AMPs. Then we
do a SORT to order Spool 6 by row hash and the sort key in
spool field1 eliminating duplicate rows. The size of Spool 6
is estimated with low confidence to be 989,301 rows (
28,689,729 bytes). The estimated time for this step is 1
minute and 36 seconds.
2) We do an all-AMPs RETRIEVE step from DB.D by way of an
all-rows scan with no residual conditions into Spool 7
(all_amps), which is built locally on the AMPs. Then we do a
SORT to order Spool 7 by the hash code of (
DB.D.DIM_COL). The size of Spool 7 is
estimated with low confidence to be 6,118,545 rows (
177,437,805 bytes). The estimated time for this step is 0.11
seconds.
5) We do an all-AMPs JOIN step from Spool 6 (Last Use) by way of an
all-rows scan, which is joined to Spool 7 (Last Use) by way of an
all-rows scan. Spool 6 and Spool 7 are joined using an exclusion
merge join, with a join condition of ("Field_1 = Field_1"). The
result goes into Spool 1 (all_amps), which is built locally on the
AMPs. The size of Spool 1 is estimated with low confidence to be
494,651 rows (14,344,879 bytes). The estimated time for this step
is 3.00 seconds.
6) We execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step from Spool 1 (Last Use) by
way of an all-rows scan into Spool 5 (all_amps), which is
redistributed by the hash code of (
DB.F.SKEW_COL) to all AMPs. Then we
do a SORT to order Spool 5 by row hash. The size of Spool 5
is estimated with low confidence to be 494,651 rows (
12,366,275 bytes). The estimated time for this step is 0.13
seconds.
2) We do an all-AMPs RETRIEVE step from DB.FF by way of an
all-rows scan with no residual conditions into Spool 8
(all_amps) fanned out into 24 hash join partitions, which is
built locally on the AMPs. The size of Spool 8 is estimated
with high confidence to be 2,603,284,805 rows (
54,668,980,905 bytes). The estimated time for this step is
24.40 seconds.
7) We do an all-AMPs RETRIEVE step from Spool 5 (Last Use) by way of
an all-rows scan into Spool 9 (all_amps) fanned out into 24 hash
join partitions, which is duplicated on all AMPs. The size of
Spool 9 is estimated with low confidence to be 249,304,104 rows (
5,235,386,184 bytes). The estimated time for this step is 1.55
seconds.
8) We do an all-AMPs JOIN step from Spool 8 (Last Use) by way of an
all-rows scan, which is joined to Spool 9 (Last Use) by way of an
all-rows scan. Spool 8 and Spool 9 are joined using a inclusion
hash join of 24 partitions, with a join condition of (
"SKEW_COL = SKEW_COL"). The
result goes into Spool 4 (all_amps), which is built locally on the
AMPs. The size of Spool 4 is estimated with index join confidence
to be 1,630,304,007 rows (37,496,992,161 bytes). The estimated
time for this step is 11.92 seconds.
9) We do an all-AMPs SUM step to aggregate from Spool 4 (Last Use) by
way of an all-rows scan , grouping by field1 (
DB.FF.SKEW_COL). Aggregate Intermediate
Results are computed globally, then placed in Spool 11. The size
of Spool 11 is estimated with low confidence to be 494,651 rows (
14,344,879 bytes). The estimated time for this step is 35.00
seconds.
10) We do an all-AMPs RETRIEVE step from Spool 11 (Last Use) by way of
an all-rows scan into Spool 2 (group_amps), which is built locally
on the AMPs. The size of Spool 2 is estimated with low confidence
to be 494,651 rows (16,323,483 bytes). The estimated time for
this step is 0.01 seconds.
11) Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> The contents of Spool 2 are sent back to the user as the result of
statement 1. The total estimated time is 2 minutes and 52 seconds.
There are some 900K unique values of skewed_ column and * ( interestingly there are 6 Million unique values for DIM_COL, which is why I think it is veering towards the Fact table column. But still..it knows from the Low Unique value in the bigger table, that its badly skewed )
My Q is after knowing that SKEWED_COL is 99% skewed due to a constant value like -9999 WHY does the optimizer still redistribute by this skewed column instead of using alternate PRPD approach. A similar ( but not same ) situation happened in past but when we upgraded to faster box ( more AMPS ) it went away .
Anything that comes to mind that will make it change plans. I tried most diagnostics - no result. Created a SI ( On a similar VT but it will still skew ).SKEWING is inevitable , ( You can artificially change the data - I am aware so to minimize this BUT all that is NOT after the fact. Now we are in PROD. Everything is over ) but even after it knows the Col is Skewed, why re-distribute it when other options are available
Its not the NULL value that skewing . Its a constant flag value ( probably value rep. of the NULL like -9999 that is causing the skew as I mentioned in the poster ) . If you rewrite the Q as I updated it works fine. I preferred NOT EXISTS because the latter will not need NULL CHECKING ( as a practice though from my DD knowledge - i know both cols are declared NOT NULL ) . I have updated the Poster with an alternative code that will work ( though like I explained - i finalized with the NOT exists version)
Select count(*) , f.SKEW_COL
from (
select ff.SKEW_COL
from DB.10_BILLON_FACT ff
where ff.SKEW_COL not in (
select d.DIM_COL
from DB.Smaller_DIM d )) as f
Group by f.SKEW_COL
Can I not get the optimizer query rewrite feature to think through the Q and rewrite with above logic. The above will NOT redistribute but JUST SORT By the Skewed Column
Until you can replace the SQL, adding spool may be your only option.
Make sure your stats are current or consider a join index with an alternative PI that covers this particular query without having to do the redistribution. You may have a skewed JI but if the work can be done AMP local you may be able to address the spool issue.
I would appreciate it if you can help me with a problem that i have.
I have this join condition :
SELECT *
FROM
T1_STAGING.(first_table) AS STG
JOIN T1_STAGING.(second_table) AS B
ON
(
STG.DLOF_ID_NO=B.DLOF_ID_NO_RU
)
This simple join is taking too long to finish, more than 20 minutes. The data of each table is less than 600,000K data. i tried the following things :
I took statistics on each table.
I changed the columns to be PRIMARY INDEX.
I created JOIN INDEX for the second table but still nothing!
The query never ends it takes 20 mins ++. This seems to be data distribution problem in the second table, but i can't do anything with the data.
Please bear in mind that if i join my first_table with any other it takes only seconds.
Can you give me a suggestion to try? I need to optimize it for better performance.
Here is the explain of TERADATA:
Explain SEL *
FROM
T1_STAGING.DLS_DLO_OWS_STAGE_STG AS STG
JOIN T1_STAGING.DLS_ACQUISITION_STG AS B
ON
(
STG.DLOF_ID_NO=B.DLOF_ID_NO_RU
)
1) First, we lock a distinct T1_STAGING."pseudo table" for read on a
RowHash to prevent global deadlock for T1_STAGING.STG.
2) Next, we lock a distinct T1_STAGING."pseudo table" for read on a
RowHash to prevent global deadlock for T1_STAGING.B.
3) We lock T1_STAGING.STG for read, and we lock T1_STAGING.B for read.
4) We execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step from T1_STAGING.B by way of
an all-rows scan with no residual conditions split into Spool
2 (all_amps) with a condition of ("DLOF_ID_NO_RU IN (:)") to
qualify skewed rows and Spool 3 (all_amps) with a condition
of ("DLOF_ID_NO_RU IN (:)") to qualify rows matching skewed
rows of the skewed relation and Spool 4 (all_amps) with
remaining rows fanned out into 2 hash join partitions. Spool
2 is built locally on the AMPs. Then we do a SORT to order
Spool 2 by row hash. The size of Spool 2 is estimated with
high confidence to be 303 rows. Spool 3 is built locally on
the AMPs. The size of Spool 3 is estimated with high
confidence to be 4,710 rows. Spool 4 is redistributed by
hash code to all AMPs. The size of Spool 4 is estimated with
high confidence to be 97,742 rows. The estimated time for
this step is 1.27 seconds.
2) We do an all-AMPs RETRIEVE step from T1_STAGING.STG by way of
an all-rows scan with no residual conditions split into Spool
6 (all_amps) with a condition of ("DLOF_ID_NO IN (:)") to
qualify skewed rows and Spool 5 (all_amps) with a condition
of ("DLOF_ID_NO IN (:)") to qualify rows matching skewed
rows of the skewed relation and Spool 7 (all_amps) with
remaining rows fanned out into 2 hash join partitions. Spool
6 is built locally on the AMPs. The size of Spool 6 is
estimated with high confidence to be 21,587 rows. Spool 5 is
built locally on the AMPs. The size of Spool 5 is estimated
with high confidence to be 7 rows. Spool 7 is redistributed
by hash code to all AMPs. The size of Spool 7 is estimated
with high confidence to be 301,682 rows. The estimated time
for this step is 4.20 seconds.
5) We execute the following steps in parallel.
1) We do an all-AMPs RETRIEVE step from Spool 5 (Last Use) by
way of an all-rows scan into Spool 8 (all_amps), which is
duplicated on all AMPs. Then we do a SORT to order Spool 8
by the hash code of (T1_STAGING.STG.DLOF_ID_NO). The size of
Spool 8 is estimated with high confidence to be 336 rows (
640,080 bytes). The estimated time for this step is 0.01
seconds.
2) We do an all-AMPs RETRIEVE step from Spool 3 (Last Use) by
way of an all-rows scan into Spool 9 (all_amps), which is
duplicated on all AMPs. The result spool file will not be
cached in memory. The size of Spool 9 is estimated with high
confidence to be 226,080 rows (391,796,640 bytes). The
estimated time for this step is 1.05 seconds.
6) We do an all-AMPs JOIN step from Spool 8 (Last Use) by way of a
RowHash match scan, which is joined to Spool 2 (Last Use) by way
of a RowHash match scan. Spool 8 and Spool 2 are joined using a
merge join, with a join condition of ("DLOF_ID_NO = DLOF_ID_NO_RU").
The result goes into Spool 1 (group_amps), which is built locally
on the AMPs. The result spool file will not be cached in memory.
The size of Spool 1 is estimated with low confidence to be 2,121
rows (11,491,578 bytes). The estimated time for this step is 0.03
seconds.
7) We do an all-AMPs JOIN step from Spool 6 (Last Use) by way of an
all-rows scan, which is joined to Spool 9 (Last Use) by way of an
all-rows scan. Spool 6 and Spool 9 are joined using a single
partition hash join, with a join condition of ("DLOF_ID_NO =
DLOF_ID_NO_RU"). The result goes into Spool 1 (group_amps), which
is built locally on the AMPs. The result spool file will not be
cached in memory. The size of Spool 1 is estimated with low
confidence to be 9,243,161 rows (50,079,446,298 bytes). The
estimated time for this step is 0.60 seconds.
8) We do an all-AMPs JOIN step from Spool 4 (Last Use) by way of an
all-rows scan, which is joined to Spool 7 (Last Use) by way of an
all-rows scan. Spool 4 and Spool 7 are joined using a hash join
of 2 partitions, with a join condition of ("DLOF_ID_NO =
DLOF_ID_NO_RU"). The result goes into Spool 1 (group_amps), which
is built locally on the AMPs. The result spool file will not be
cached in memory. The size of Spool 1 is estimated with low
confidence to be 731,525 rows (3,963,402,450 bytes). The
estimated time for this step is 0.96 seconds.
9) Finally, we send out an END TRANSACTION step to all AMPs involved
in processing the request.
-> The contents of Spool 1 are sent back to the user as the result of
statement 1. The total estimated time is 6.84 seconds.
600,000K you mean 600M, right? ;) That's not that little.
First. The columns used in JOIN should be indexed. (Looks like you've done it)
2. Add WHERE condition to choose specific values you need
3. Add LIMIT to limit the SELECT result
And finally, use EXPLAIN SELECT ... to understand the issue.