Related
We have a select statement in production that takes quite a lot of time.
The current query uses row number - window function.
I am trying to rewrite the query and test the same. assuming its orc table fetching aggregate values instead of using row number may help to reduce the execution time, is my assumption
Is something like this possible. Let me know if i am missing anything.
Sorry i am trying to learn, so please bear with my mistakes, if any.
I tried to rewrite the query as mentioned below.
Original query
SELECT
Q.id,
Q.crt_ts,
Q.upd_ts,
Q.exp_ts,
Q.biz_effdt
(
SELECT u.id, u.crt_ts, u.upd_ts, u.exp_ts, u.biz_effdt, ROW_NUMBER() OVER (PARTITION BY u.id ORDER BY u.crt_ts DESC) AS ROW_N
FROM ( SELECT cust_prd.id, cust_prd.crt_ts, cust_prd.upd_ts, cust_prd.exp_ts, cust_prd.biz_effdt FROM MSTR_CORE.cust_prd
WHERE biz_effdt IN ( SELECT MAX(cust_prd.biz_effdt) FROM MSTR_CORE.cust_prd )
) U
)Q WHERE Q.row_n = 1
My attempt:
SELECT cust_prd.id, cust_prd.crt_ts, cust_prd.upd_ts, cust_prd.exp_ts, cust_prd.biz_effdt FROM MSTR_CORE.cust_prd
WHERE biz_effdt IN ( SELECT MAX(cust_prd.biz_effdt) FROM MSTR_CORE.cust_prd )
having cust_prd.crt_ts = max (cust_prd.crt_ts)
The problem I'm facing right now is I'm working with a SQL query that has over 200 lines of code and at the moment in multiple cases I'm just repeating the same sub-query multiple times in this select statement. In the code below I'm using two of the select statements a lot "avail_qty" and "pct_avail" which both having equations in them. Inside the LOW_CNT_&% SELECT statement I use both of the previous two SELECT statements over and over (this is just one example in my code). I would like to be able to make the equation once and assign it to a variable. Is there any way of doing this? I have tried using the WITH clause but for that you need to use a FROM clause, my FROM clause is massive and would look just as ugly if I were to use a WITH clause (plus instead of repeating the SELECT statement now I would be just repeating the FROM statement).
The reason I don't want to type out the whole equation multiple times is for a two reasons the first is it makes the code easier to read. My other reason is because multiple people edit this query and if someone else were to edit the equation in one spot but forgets to edit it in another spot, that could be bad. Also it doesn't feel like good code etiquette to repeat code over and over.
SELECT
all_nbr.total_qty,
NVL (avail_nbr.avail_qty, 0) AS avail_qty,
100 * TRUNC ( (NVL (avail_nbr.avail_qty, 0) / all_nbr.total_qty), 2) AS pct_avail,
CASE
WHEN ((NVL (avail_nbr.avail_qty, 0)) < 35)
THEN CASE
WHEN ((100 * TRUNC ( (NVL (avail_nbr.avail_qty, 0) / all_nbr.total_qty), 2)) < 35)
THEN (35 - (NVL (avail_nbr.avail_qty, 0)))
ELSE 0
END
ELSE 0
END AS "LOW_CNT_&%"
FROM
...
Any help would be awesome!!
If the subquery is exactly the same one, you can pre-compute it as a Common Table Expression (CTE). For example:
with
cte1 as (
select ... -- long, tedious, repetitive SELECT here
),
cte2 as (
select ... -- you can reference/use cte1 here
)
select ...
from cte1 -- you can use cte1 here, multiple times if you want
join cte2 -- you can also reference/use cte2 here, also multiple times
join ... -- all other joins
cte1 (you can use any name) is a precomputed table expression that can be used multiple times. You can also have multiple CTEs, each one with different names; also each CTE can reference previous ones.
I have tried using the WITH clause but for that you need to use a FROM clause, my FROM clause is massive and would look just as ugly if I were to use a WITH clause (plus instead of repeating the SELECT statement now I would be just repeating the FROM statement).
You shouldn't need to repeat the from clause. You move all of the query, including that clause, into the CTE; you just pull out the bits that rely on earlier calculations into the main query, which avoids the code repetition.
The structure would be something like:
WITH cte AS (
SELECT
all_nbr.total_qty,
NVL (avail_nbr.avail_qty, 0) AS avail_qty,
100 * TRUNC ( (NVL (avail_nbr.avail_qty, 0) / all_nbr.total_qty), 2) AS pct_avail,
FROM
...
)
SELECT
cte.total_qty,
cte.avail_qty,
cte.pct_avail,
CASE
WHEN cte.avail_qty, 0 < 35
THEN CASE
WHEN cte.total_qty < 35
THEN 35 - cte.avail_qty
ELSE 0
END
ELSE 0
END AS "LOW_CNT_&%"
FROM
cte;
Your main query only need to refer to the CTE (again, based on what you've shown), and can (only) refer to the prjoection of the CTE, incuding the calculated columns. It can't see the underlying tables, but shouldn't need to.
Or with an inline view instead, the principal is the same:
SELECT
total_qty,
avail_qty,
pct_avail,
CASE
WHEN avail_qty < 35
THEN CASE
WHEN total_qty < 35
THEN 35 - avail_qty
ELSE 0
END
ELSE 0
END AS "LOW_CNT_&%"
FROM
(
SELECT
all_nbr.total_qty,
NVL (avail_nbr.avail_qty, 0) AS avail_qty,
100 * TRUNC ( (NVL (avail_nbr.avail_qty, 0) / all_nbr.total_qty), 2) AS pct_avail,
FROM
...
);
Im looking for something like SELECT PRODUCT(table.price) FROM table GROUP BY table.sale similar to how SUM works.
Have I missed something on the documentation, or is there really no PRODUCT function?
If so, why not?
Note: I looked for the function in postgres, mysql and mssql and found none so I assumed all sql does not support it.
For MSSQL you can use this. It can be adopted for other platforms: it's just maths and aggregates on logarithms.
SELECT
GrpID,
CASE
WHEN MinVal = 0 THEN 0
WHEN Neg % 2 = 1 THEN -1 * EXP(ABSMult)
ELSE EXP(ABSMult)
END
FROM
(
SELECT
GrpID,
--log of +ve row values
SUM(LOG(ABS(NULLIF(Value, 0)))) AS ABSMult,
--count of -ve values. Even = +ve result.
SUM(SIGN(CASE WHEN Value < 0 THEN 1 ELSE 0 END)) AS Neg,
--anything * zero = zero
MIN(ABS(Value)) AS MinVal
FROM
Mytable
GROUP BY
GrpID
) foo
Taken from my answer here: SQL Server Query - groupwise multiplication
I don't know why there isn't one, but (take more care over negative numbers) you can use logs and exponents to do:-
select exp (sum (ln (table.price))) from table ...
There is no PRODUCT set function in the SQL Standard. It would appear to be a worthy candidate, though (unlike, say, a CONCATENATE set function: it's not a good fit for SQL e.g. the resulting data type would involve multivalues and pose a problem as regards first normal form).
The SQL Standards aim to consolidate functionality across SQL products circa 1990 and to provide 'thought leadership' on future development. In short, they document what SQL does and what SQL should do. The absence of PRODUCT set function suggests that in 1990 no vendor though it worthy of inclusion and there has been no academic interest in introducing it into the Standard.
Of course, vendors always have sought to add their own functionality, these days usually as extentions to Standards rather than tangentally. I don't recall seeing a PRODUCT set function (or even demand for one) in any of the SQL products I've used.
In any case, the work around is fairly simple using log and exp scalar functions (and logic to handle negatives) with the SUM set function; see #gbn's answer for some sample code. I've never needed to do this in a business application, though.
In conclusion, my best guess is that there is no demand from SQL end users for a PRODUCT set function; further, that anyone with an academic interest would probably find the workaround acceptable (i.e. would not value the syntactic sugar a PRODUCT set function would provide).
Out of interest, there is indeed demand in SQL Server Land for new set functions but for those of the window function variety (and Standard SQL, too). For more details, including how to get involved in further driving demand, see Itzik Ben-Gan's blog.
You can perform a product aggregate function, but you have to do the maths yourself, like this...
SELECT
Exp(Sum(IIf(Abs([Num])=0,0,Log(Abs([Num])))))*IIf(Min(Abs([Num]))=0,0,1)*(1-2*(Sum(IIf([Num]>=0,0,1)) Mod 2)) AS P
FROM
Table1
Source: http://productfunctionsql.codeplex.com/
There is a neat trick in T-SQL (not sure if it's ANSI) that allows to concatenate string values from a set of rows into one variable. It looks like it works for multiplying as well:
declare #Floats as table (value float)
insert into #Floats values (0.9)
insert into #Floats values (0.9)
insert into #Floats values (0.9)
declare #multiplier float = null
select
#multiplier = isnull(#multiplier, '1') * value
from #Floats
select #multiplier
This can potentially be more numerically stable than the log/exp solution.
I think that is because no numbering system is able to accommodate many products. As databases are designed for large number of records, a product of 1000 numbers would be super massive and in case of floating point numbers, the propagated error would be huge.
Also note that using log can be a dangerous solution. Although mathematically log(a*b) = log(a)*log(b), it might not be in computers as we are not dealing with real numbers. If you calculate 2^(log(a)+log(b)) instead of a*b, you may get unexpected results. For example:
SELECT 9999999999*99999999974482, EXP(LOG(9999999999)+LOG(99999999974482))
in Sql Server returns
999999999644820000025518, 9.99999999644812E+23
So my point is when you are trying to do the product do it carefully and test is heavily.
One way to deal with this problem (if you are working in a scripting language) is to use the group_concat function.
For example, SELECT group_concat(table.price) FROM table GROUP BY table.sale
This will return a string with all prices for the same sale value, separated by a comma.
Then with a parser you can get each price, and do a multiplication. (In php you can even use the array_reduce function, in fact in the php.net manual you get a suitable example).
Cheers
Another approach based on fact that the cardinality of cartesian product is product of cardinalities of particular sets ;-)
⚠ WARNING: This example is just for fun and is rather academic, don't use it in production! (apart from the fact it's just for positive and practically small integers)⚠
with recursive t(c) as (
select unnest(array[2,5,7,8])
), p(a) as (
select array_agg(c) from t
union all
select p.a[2:]
from p
cross join generate_series(1, p.a[1])
)
select count(*) from p where cardinality(a) = 0;
The problem can be solved using modern SQL features such as window functions and CTEs. Everything is standard SQL and - unlike logarithm-based solutions - does not require switching from integer world to floating point world nor handling nonpositive numbers. Just number rows and evaluate product in recursive query until no row remain:
with recursive t(c) as (
select unnest(array[2,5,7,8])
), r(c,n) as (
select t.c, row_number() over () from t
), p(c,n) as (
select c, n from r where n = 1
union all
select r.c * p.c, r.n from p join r on p.n + 1 = r.n
)
select c from p where n = (select max(n) from p);
As your question involves grouping by sale column, things got little bit complicated but it's still solvable:
with recursive t(sale,price) as (
select 'multiplication', 2 union
select 'multiplication', 5 union
select 'multiplication', 7 union
select 'multiplication', 8 union
select 'trivial', 1 union
select 'trivial', 8 union
select 'negatives work', -2 union
select 'negatives work', -3 union
select 'negatives work', -5 union
select 'look ma, zero works too!', 1 union
select 'look ma, zero works too!', 0 union
select 'look ma, zero works too!', 2
), r(sale,price,n,maxn) as (
select t.sale, t.price, row_number() over (partition by sale), count(1) over (partition by sale)
from t
), p(sale,price,n,maxn) as (
select sale, price, n, maxn
from r where n = 1
union all
select p.sale, r.price * p.price, r.n, r.maxn
from p
join r on p.sale = r.sale and p.n + 1 = r.n
)
select sale, price
from p
where n = maxn
order by sale;
Result:
sale,price
"look ma, zero works too!",0
multiplication,560
negatives work,-30
trivial,8
Tested on Postgres.
Here is an oracle solution for anyone who needs it
with data(id, val) as(
select 1,1.0 from dual union all
select 2,-2.0 from dual union all
select 3,1.0 from dual union all
select 4,2.0 from dual
),
neg(val , modifier) as(
select exp(sum(ln(abs(val)))), case when mod(count(*),2) = 0 then 1 Else -1 end
from data
where val <0
)
,
pos(val) as (
select exp(sum(ln(val)))
from data
where val >=0
)
select (select val*modifier from neg)*(select val from pos) product from dual
I have two subquerys both calculating sums. I would like to do an Artithmetic Minus(-) with the result of both Querys . eg Query1: 400 Query2: 300 Result should be 100.
Obvious a basic - in the query does not work. The minus works as MINUS on sets. How can I solve this? Do you have any ideas?
SELECT CustumersNo FROM Custumers WHERE
(
SELECT SUM(value) FROM roe WHERE roe.credit = Custumers.CustumersNo
-
SELECT SUM(value) FROM roe WHERE roe.debit = Custumers.CustumersNo
)
> 500
Using Informix - sorry missed that point
To get the original syntax to work, you would need to surround the sub-selects in parentheses:
SELECT CustumersNo
FROM Custumers
WHERE ((SELECT SUM(value) FROM roe WHERE roe.credit = Custumers.CustumersNo)
-
(SELECT SUM(value) FROM roe WHERE roe.debit = Custumers.CustumersNo)
) > 500
Note that aggregates are defined to ignore nulls in the values they aggregate in standard SQL. However, the SUM of an empty set of rows is NULL, not zero.
You can get inventive and devise ways to always have a value for each customer listed in the roe table, such as:
SELECT CustomersNo
FROM (SELECT CustomersNo, SUM(value) AS net_credit
FROM (SELECT credit AS CustomersNo, +value
UNION
SELECT debit AS CustomersNo, -value
) AS x
GROUP BY CustomersNo
) AS y
WHERE net_credit > 500;
You can also do that with an appropriate HAVING clause if you wish. Note that this avoids issues with customers who have credit entries but no debit entries or vice versa; all the entries that are present are treated appropriately.
Your misspelling (or unorthodox spelling) of 'customers' is nearly as good as 'costumers'.
Something like what you tried should work. It may be a syntax problem, and it may depend on what type of SQL you are using. However, an approach like this would be more efficient:
Update: I see you were having a problem with nulls, so I updated it to handle nulls properly.
select CustumersNo from (
select CustumersNo,
sum(coalesce(roecredit.value,0)) - sum(coalesce(roedebit.value,0))
as balance
FROM Custumers
join roe roecredit on roe.credit = Custumers.CustumersNo
join roe roedebit on roe.debit = Custumers.CustumersNo
group by CustumersNo
)
where balance > 500
Caveat: I don't have experience with Informix specifically.
I have the following query:
SELECT c004, mesosafe, CAST(SUBSTRING (c000 ,1 , 5) as int) as sortorder, c001, c000, c002
FROM DE_DATA.dbo.t309
WHERE c001 IS NOT NULL
AND c002 = 1
AND mesocomp = 'EMTD'
AND mesoyear IN (
SELECT MAX(mesoyear) AS currmesoyear
FROM DE_DATA.dbo.t001
)
AND CAST(SUBSTRING (c000 ,1 , 5) as int) < 101
ORDER BY c000 ASC;
This query fails because some values of c000 cannot be casted because they look like 008- instead of 00052-. Here the mesoyear is asked (mesoyear IN ...). If I query this SQL part alone I get 1344 as result.
On the other side this query works:
SELECT c004, mesosafe, CAST(SUBSTRING (c000 ,1 , 5) as int) as sortorder, c001, c000, c002
FROM DE_DATA.dbo.t309
WHERE c001 IS NOT NULL
AND c002 = 1
AND mesocomp = 'EMTD'
AND mesoyear IN (
'1344'
)
AND CAST(SUBSTRING (c000 ,1 , 5) as int) < 101
ORDER BY c000 ASC;
So what is the difference between hardcoded values and the SQL query?
Edit:
I think the reason is that the subquery is evaluated later than the main query. Can it be? What can I do against this?
You are assuming a certain order of execution, in as only certain rows you believe are 'correct' will be evaluated against the CAST operation. This is a fundamental fallacy. SQL is a declarative, set oriented language which does not make any evaluation ordering promise the way imperative languages do. As such your entire approach is flawed and you're asking the wrong question. Some query execution plans may work, some may fail, but those that work will start failing randomly later as the query chooses a different plan.
Ultimately your problem is the data model, the fact that you have to crack this composite c000 field into substrings and cast to get out int values. Use string fields to store strings, use numeric fields to store numbers. Simple as that. What you're trying to achieve will never work.
See also On SQL Server boolean operator short-circuit and T-SQL Functions do not imply a certain order of execution.
Usually MsSql runs subqueries before their parent. You query probably fails when the inner query finds the first value it cannot cast to the same type-and-size of mesoyear. Did you check if columns t309.mesoyear and t001.mesoyear have the same type?
Nevertheless, there are some cases when MsSql doesn't follow the usual behaviour. If you think this is the case, consider using the FORCE ORDER query hint (although this situation will eventually emerge again).
Last, but not least, if the CAST operator is not safe, you can (should) check if the value can be cast before incurring into a runtime error:
-- use
AND (1=1
AND (ISNUMERIC(SUBSTRING(c000 ,1 , 5) = 1)
AND (CAST(SUBSTRING (c000 ,1 , 5) as int) < 101)
)
-- instead of
AND CAST(SUBSTRING (c000 ,1 , 5) as int) < 101