I want to optimize a select query in a loop of a SQL procedure. The loop iterates around 10000 times and the select query takes approx. 30 ms for each iteration which increases the overall execution time of the procedure
SELECT *
FROM BANKACCOUNTS B,
MAPPING M,
UPL_DTR_UPLOAD UP,
(SELECT * FROM MAPPING WHERE SOURCE = 'KARVY_BANK_CODE') M1
WHERE B.SCHEME_CODE = M.INTERNALCODE
AND M1.INTERNALCODE = B.BANK_CODE
AND M.SOURCE = 'R0'
AND B.AC_TYPE = 'FUNDING'
AND M.EXTERNALCODE IS NOT NULL
AND UPPER(TRIM(M.EXTERNALCODE || M1.EXTERNALCODE || B.AC_NO)) =
Upper(UP.Scheme || UP.Fundingbnk || UP.fundingacc);
There are lot of solutions
But first use modern,explicit joins.
Your query for column m1 contains *, use required columns only
Check explain plan and use of index
Code:
SELECT *
FROM bankaccounts B
JOIN mapping M ON B.scheme_code = M.internalcode
JOIN
(SELECT internalcode, externalcode
FROM mapping
WHERE source = 'KARVY_BANK_CODE') M1 ON M1.internalcode = B.bank_code
JOIN upl_dtr_upload UP ON UPPER(TRIM(M.externalcode || M1.externalcode || B.ac_no)) = UPPER(UP.scheme || UP.fundingbnk || UP.fundingacc)
WHERE
M.source = 'R0'
AND B.ac_type = 'FUNDING'
AND M.externalcode IS NOT NULL;
As #LoztInSpace mentions, you can almost certainly replace your PL/SQL loop to iterate "about 10,000 times" to become the driving query. IE: if you need to do something with the results from each row returned in your query you posted, for each row in the "do something about 10,000 times" that implies the outside loop is another query, then nest your query (well, Kedar's version of the query) inside your outer loop.
Each execution of the PL/SQL loop is going to have to invoke the SQL engine, forcing a context switch; that is probably 10 ms of your 30 ms if not more. Search https://asktom.oracle.com with keywords PL/SQL "nested loop" for examples.
You can also look at PL/SQL bulk processing statements FORALL and BULK COLLECT for possible improvements.
Related
I am stuck in a difficult situation. I need to compare the counts of 2 and update one of the tables based on the comparison. Below are the details:
I have 2 tables: dwh_idh_to_iva_metadata_t and lnd_sml_price_t.
Every day there is a package running that loads data from source table dwh_sml_price_t to destination table lnd_sml_price_t. This package also checks the counts between the tables "dwh_idh_to_iva_metadata_t" and "lnd_sml_price_t". If the counts are matching then it updates few columns in the table dwh_idh_to_iva_metadata_t and if the counts are not matching then the package throws an exception and exits. To do the count comparison we created a cursor and then fetching that cursor for comparison. The code for cursor is as follows:
CURSOR C_CNT_PRICE IS
SELECT
lnd.smp_batchrun_id batch_id,
lnd.lnd_count,
dwh.dwh_count
FROM
(
SELECT
smp_batchrun_id,
COUNT(*) lnd_count
FROM
iva_landing.lnd_sml_price_t
GROUP BY
smp_batchrun_id
) lnd
LEFT JOIN (
SELECT
batchrun_id,
sent_records_count dwh_count,
dwh_sending_table
FROM
dwh.dwh_idh_to_iva_metadata_t
) dwh ON dwh.batchrun_id = lnd.smp_batchrun_id
WHERE
dwh.dwh_sending_table = 'DWH_SML_PRICE_T'
ORDER BY
1 DESC; `
And the actual code for comparison is:
` FOR L_COUNT IN C_CNT_PRICE LOOP --0001,0002
IF L_COUNT.lnd_count = L_COUNT.dwh_count THEN
UPDATE DWH.DWH_IDH_TO_IVA_METADATA_T idh
SET idh.IVA_RECEIVING_TABLE = 'LND_SML_PRICE_T',
idh.RECEIVED_DATE = SYSDATE,
idh.RECEIVED_RECORDS_COUNT = L_COUNT.lnd_count,
idh.status = 'Verified'
WHERE L_COUNT.batch_id = idh.batchrun_id
AND idh.dwh_sending_table = 'DWH_SML_PRICE_T';
COMMIT;
ELSE
RAISE EXCPT_MISSDATA; -- Throw error and exit process immediately
END IF;
END LOOP; `
Now, the problem is that there may be certain cases when the table "LND_SML_PRICE_T" is not having any data, in which case, DWH_IDH_TO_IVA_METDATA_T should have the columns updated as in case of matching counts.
I need help in modifying the code so that comparison is done even in case of no records in LND_SML_PRICE_T table.
Thanks!
If there are no rows in LND_SML_PRICE_T table, cursor C_CNT_PRICE's LND_COUNT equals 0 which means that comparison code should be modified to
FOR L_COUNT IN C_CNT_PRICE LOOP
IF (L_COUNT.lnd_count = L_COUNT.dwh_count)
or L_COUNT.lnd_count = 0 --> this
THEN
I have this below Oracle query. It takes ages to execute.
Select Distinct Z.WH_Source,
substr(Z.L_Y_Month,0,4) || '-' || substr(Z.L_Y_Month,5) Ld_Yr_Mth,
m.model_Name, p.SR, p.PLATE_NO, pp.value, z.CNT_number, z.platform_SR_number,
z.account_name, z.owner_name, z.operator_name, z.jetcare_expiry_date, z.wave,
z.address, z.country, substr(z.CNT_status, 10) ctstatus,
ALLOEM.GET_CNT_TYRE_SR#TNS_GG(z.CNT_number, Z.WH_Source, Z.L_Y_Month,
z.platform_SR_number, '¿')
product_SR_number
From MST.ROLE p
inner join MST.model m on m.model_id = p.model_id
left join MST.ROLEproperty pp on pp.ROLE_id = p.ROLE_id
and pp.property_lookup = 'SSG-WH-ENROLL'
left join alloem.Z_SSG_HM_LOG#TNS_GG z on z.camp_ac_ROLE_id = p.ROLE_id
Where
1 = 1 or z.L_Y_Month = 1
Order By 1, 2 desc, 3,4
If i remove this line,
ALLOEM.GET_CNT_TYRE_SR#TNS_GG(z.CNT_number, Z.WH_Source, Z.L_Y_Month,
z.platform_SR_number, '¿')
it executes very fast. But, I can't remove the line. Is there any way to make this query to execute fast.?
If i remove this line,
ALLOEM.GET_CNT_TYRE_SR#TNS_GG(z.CNT_number, Z.WH_Source,
Z.L_Y_Month, z.platform_SR_number, '¿')
it executes very fast. But, I can't remove the line. Is there any way to make this query to execute fast.?
Query tuning is a complex thing. Without table structures, indexes, execution plan or statistics it is very hard to provide one universal answer.
Anyway I would try scalar subquery caching(if applicable):
ALLOEM.GET_CNT_TYRE_SR#TNS_GG(z.CNT_number, Z.WH_Source, Z.L_Y_Month,
z.platform_SR_number, '¿')
=>
(SELECT ALLOEM.GET_CNT_TYRE_SR#TNS_GG(z.CNT_number, Z.WH_Source,Z.L_Y_Month,
z.platform_SR_number, '¿') FROM dual)
Also usage of DISTINCT may indicate some problems with normalization. If possible please fix underlying problem and remove it.
Finally you should avoid using positional ORDER BY (it is commom anti-pattern).
This:
alloem.Z_SSG_HM_LOG#TNS_GG
suggests that you fetch data over a database link. It is usually slower than fetching data locally. So, if you can afford it & if your query manipulates "static" data (i.e. nothing changes in Z_SSG_HM_LOG table frequently) and - even if it does - the amount of data isn't very high, consider creating a materialized view (MV) in schema you're connected to while running that query. You can even create index(es) on a MV so ... hopefully, everything will run faster without too much effort.
I'm trying to understand this behaviour. My below statement takes nearly half an hour to complete. However when I replace the parameter #IsGazEnabled (in the case statement of the where clause at the bottom) with a value of 1, then it takes just a second.
Looking at the estimated execution plan, when using the parameter and when it takes 30 minutes, most of the cost (92%) lies with a Nested Loop (Left Anti Semi Join). there also seems to be some Parallelism going on. I'm only just learning about Execution plans and I'm left a bit confused. Very different to the plan produced when not using the parameter.
So how does having a parameter compared to not having one make such a difference to the execution plan and performance?
declare #IsGazEnabled tinyint;
set #IsGazEnabled = 1;
select 'CT Ref: ' + accountreference + ' - Not synced due to missing property ref ' + t.PropertyReference
from CTaxAccountTemp t
where not exists (
select *
from ccaddress a
left join w2addresscrossref x on x.UPRN = a.UPRN
and x.appcode in (
select w2source
from GazSourceConfig
where GazSource in (
select GazSource
from GazSourceConfig
where W2Source = 'CTAX'
)
union all select 'URB'
)
where t.PropertyReference = case #IsGazEnabled when 1 then x.PropertyReference else a.PropertyReference end
);
This can occur because SQL Server (the query optimizer) uses the value of the provided parameter when creating the initial execution plan. If the values in some of your tables are not evenly distributed, the created plan may work really well for certain values of the parameter, but really poorly for others. This is generally referred to a parameter sniffing. You can get around this using query hints (OPTIMIZE FOR X) or recompiling the stored procedure before each run (WITH RECOMPILE). You should read up on these options thoroughly before implementing as they both have side effects.
See a couple articles on Brent Ozar's site for more details, here and here.
I think you should rethink this query. Try and avoid the NOT EXISTS() for starters - as thats generally quite inefficient (I usually prefer a LEFT JOIN in these instances - and a corresponding WHERE x IS NULL - the x being something in the right hand side)
The main cause of woe for you though is likely to be the CASE based WHERE - as that is now causing the inner query to be evaluated for EVERY ROW!. I think you'd be better left joining both sets of disqualifying criteria, but include the parameter in the join conditions - and then check that there is nothing on the right hand side of either of the 2 left joined criteria
Heres how I think it could be rewritten:
declare #IsGazEnabled tinyint;
set #IsGazEnabled = 1;
select 'CT Ref: ' + accountreference + ' - Not synced due to missing property ref ' + t.PropertyReference
from CTaxAccountTemp t
left join ccaddress a2 ON t.PropertyReference = a2.PropertyReference and #IsGazEnabled = 0
left join
(
ccaddress a
join w2addresscrossref x on x.UPRN = a.UPRN
and x.appcode in ( -- could make this a join for efficiency....
select w2source
from GazSourceConfig
where GazSource in (
select GazSource
from GazSourceConfig
where W2Source = 'CTAX'
)
union all select 'URB'
)
) ON t.PropertyReference = x.PropertyReference AND and #IsGazEnabled = 1
WHERE
a2.PropertyReference IS NULL
AND x.PropertyReference IS NULL
;
I have a SQL Query that comprise of two level sub-select. This is taking too much time.
The Query goes like:
select * from DALDBO.V_COUNTRY_DERIV_SUMMARY_XREF
where calculation_context_key = 130205268077
and DERIV_POSITION_KEY in
(select ctry_risk_derivs_psn_key
from DALDBO.V_COUNTRY_DERIV_PSN
where calculation_context_key = 130111216755
--and ctry_risk_derivs_psn_key = 76296412
and CREDIT_PRODUCT_TYPE = 'SWP OP'
and CALC_OBLIGOR_COUNTRY_OF_ASSETS in
(select ctry_cd
from DALDBO.V_PSN_COUNTRY
where calculation_context_key = 130134216755
--and ctry_risk_derivs_psn_key = 76296412
)
)
These tables are huge! Is there any optimizations available?
Without knowing anything about your table or view definitions, indexing, etc. I would start by looking at the sub-selects and ensuring that they are performing optimally. I would also want to know how many values are being returned by each sub-select as this can impact performance.
How is calculation_context_key used to retrieve rows from V_COUNTRY_DERIV_PSN and V_PSN_COUNTRY? Is it an optimal execution plan?
How is DERIV_POSITION_KEY and CALC_OBLIGOR_COUNTRY_OF_ASSETS used in V_COUNTRY_DERIV_SUMMARY_XREF to retrieve rows? Again, look at the explain plan.
first of all, can you write this query using inner joins (and not subselect) ??
select A.*
from DALDBO.V_COUNTRY_DERIV_SUMMARY_XREF a,
DALDBO.V_COUNTRY_DERIV_PSN b,
DALDBO.V_PSN_COUNTRY c
where calculation_context_key = 130205268077
and a.DERIV_POSITION_KEY = b.ctry_risk_derivs_psn_key
and b.calculation_context_key = 130111216755
--and b.ctry_risk_derivs_psn_key = 76296412
and b.CREDIT_PRODUCT_TYPE = 'SWP OP'
and b.CALC_OBLIGOR_COUNTRY_OF_ASSETS = c.ctry_cd
and c.calculation_context_key = 130134216755
--and c.ctry_risk_derivs_psn_key = 76296412
second, best practice says that when you don't query any data from the tables in the subselect you better of using an EXISTS instead of IN. new versions of oracle does that automatically and actually rewrite the whole thing as an inner join.
last, without any knowledge on you data and of what you are trying to do i would suggest you to try and use views as less as you can - if you can query the underling tables it would be best and you will probably see immediate performance improvement.
Im Using toad for Oracle to run a query which is taking much too long to run, sometimes over 15 minutes.
The query is pulling memos which are left to be approved by managers. The query is not bringing back alot of rows. Typically when it is run it will return about 30 or 40 rows. The query needs to access a few tables for its information so I'm using alot of joins to get this information.
I have attached my query below.
If anyone can help with optimising this query I would be very greatfull.
Query:
SELECT (e.error_Description || DECODE(t.trans_Comment, 'N', '', '','', ' - ' || t.trans_Comment)) AS Title,
t.Date_Time_Recorded AS Date_Recorded,
DECODE(t.user_ID,0,'System',(SELECT Full_Name FROM employee WHERE t.user_Id = user_id)) AS Recorded_by,
DECODE(t.user_ID,0, Dm_General.getCalendarShiftName(t.Date_Time_Recorded), (SELECT shift FROM employee WHERE t.user_Id = user_id)) AS Shift,
l.Lot_Number AS entity_number,
ms.Line_Num,
'L' AS Entity_Type,
t.entity_id, l.lot_Id AS Lot_Id
FROM DAT_TRANSACTION t
JOIN ADM_ERRORCODES e ON e.error_id = t.error_id
JOIN ADM_ACTIONS a ON a.action_id = t.action_id,
DAT_LOT l
INNER JOIN Status s ON l.Lot_Status_ID = s.Status_ID,
DAT_MASTER ms
INNER JOIN ADM_LINE LN ON ms.Line_Num = LN.Line_Num
WHERE
(e.memo_req = 'Y' OR a.memo_req = 'Y')
AND ms.Run_type_Id = Constants.Runtype_Production_Run --Production Run type
AND s.completed_type NOT IN ('D', 'C', 'R') -- Destroyed /closed / Released
AND LN.GEN = '2GT'
AND (NOT EXISTS (SELECT 1 FROM LNK_MEMO_TRANS lnk, DAT_MEMO m
WHERE lnk.Trans_ID = t.trans_id AND lnk.Memo_ID = m.Memo_ID
AND NVL(m.approve, 'Y') = 'Y'))--If it's null, it's
been created and is awaiting approval
AND l.Master_ID = ms.Master_ID
AND t.Entity_ID = l.Lot_ID
AND t.Entity_Type IN ('L', 'G');
The usual cause for bad performance of queries is that Oracle can't find an appropriate index. Use EXPLAIN PLAN with TOAD so Oracle can tell you what it thinks the best way to execute the query. That should give you some idea when it uses indexes and when not.
For general pointers, see http://www.orafaq.com/wiki/Oracle_database_Performance_Tuning_FAQ
See here for EXPLAIN PLAN.
You have some function calls in your SQL:
dm_general.getcalendarshiftname(t.date_time_recorded)
constants.runtype_production_run
Function calls are slow in SQL, and depending on the query plan may get called redundantly many times - e.g. computing dm_general.getcalendarshiftname for rows that end up being filtered out of the results.
To see if this is a significant factor, try replacing the function calls with literal constants temporarily and see if the performance improves.
The number of function calls can sometimes be reduced by restructuring the query like this:
select /*+ no_merge(v) */ a, b, c, myfunction(d)
from
( select a, b, c, d
from my_table
where ...
) v;
This ensures that myfunction is only called for rows that will appear in the results.
I have replaced function calls with literal constants and this speeds it up by only a second or 2. The query is still taking about about 50 seconds to run.
Is there anything I can do around the Joins to help spped this up. Have a used the INNER JOIN function correctly here.
Im not really sure I understand what you mean about the below or how to use it.
I get the error d invalid identifier when I try to call the function in the second select
select /*+ no_merge(v) */ a, b, c, myfunction(d)
from
( select a, b, c, d
from my_table
where ...
) v;
Any other views would be greatly appreciated
Before we can say anything sensible, we have to take a look at where time is being spent. And that means you have to collect some information first.
Therefore, my standard reaction to a question like this, is this one: http://forums.oracle.com/forums/thread.jspa?threadID=501834
Regards,
Rob.