Only Select First Reverse Duplicate - sql

I have the following table in a SQL Server 2014 database:
+----+-------+--------+---------+
| ID | CODE | NUMBER | BALANCE |
+----+-------+--------+---------+
| 1 | B0001 | 122960 | 100.00 |
+----+-------+--------+---------+
| 2 | B0001 | 123168 | -100.00 |
+----+-------+--------+---------+
| 3 | B0001 | 121400 | 500.00 |
+----+-------+--------+---------+
| 4 | T0001 | 19755 | 50.00 |
+----+-------+--------+---------+
| 5 | T0001 | 19975 | -50.00 |
+----+-------+--------+---------+
| 6 | T0001 | 122202 | 50.00 |
+----+-------+--------+---------+
| 7 | T0001 | 122203 | 50.00 |
+----+-------+--------+---------+
I am trying to select rows where the balances for a given code can be offset against another row and totaled to 0. For example, the balance on rows 1 and 2 sum to 0 so should be returned. I have tried the following query:
SELECT T1.NUMBER
FROM TABLE T1, TABLE T2
WHERE T1.CODE = T2.CODE
AND T1.BALANCE + T2.BALANCE = 0
This works OK for code B0001. It will return rows 1 and 2 which cancel each other out and ignore row 3. I'm having a problem with code T0001 because the query I'm using will match each of the 3 positive values with the negative value and return all rows associated with that code. I only want it to return rows 4 and 5 for T0001.

Try this:
/* DATASET MOCK-UP */
DECLARE #Data TABLE ( ID INT, CODE VARCHAR(10), NUMBER INT, BALANCE DECIMAL(18,2) );
INSERT INTO #Data ( ID, CODE, NUMBER, BALANCE ) VALUES
( 1, 'B0001', 122960 , 100.00 ),
( 2, 'B0001', 123168 , -100.00 ),
( 3, 'B0001', 121400 , 500.00 ),
( 4, 'T0001', 19755 , 50.00 ),
( 5, 'T0001', 19975 , -50.00 ),
( 6, 'T0001', 122202 , 50.00 ),
( 7, 'T0001', 122203 , 50.00 );
/*
Return records where combined balances equal 0 by adding the
current record's BALANCE against its previous (lag) or following (lead) balances.
*/
SELECT
ID, CODE, NUMBER, BALANCE, ( BALANCE + LAG_BALANCE ) AS LAG_BALANCE, ( BALANCE + LEAD_BALANCE ) AS LEAD_BALANCE
FROM (
SELECT
ID,
CODE,
NUMBER,
BALANCE,
LAG ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LAG_BALANCE,
LEAD ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LEAD_BALANCE
FROM #Data
) AS Results
WHERE
BALANCE + LAG_BALANCE = 0
OR
BALANCE + LEAD_BALANCE = 0
ORDER BY
ID;
Returns
+----+-------+--------+---------+-------------+--------------+
| ID | CODE | NUMBER | BALANCE | LAG_BALANCE | LEAD_BALANCE |
+----+-------+--------+---------+-------------+--------------+
| 1 | B0001 | 122960 | 100.00 | 100.00 | 0.00 |
| 2 | B0001 | 123168 | -100.00 | 0.00 | 400.00 |
| 4 | T0001 | 19755 | 50.00 | 550.00 | 0.00 |
| 5 | T0001 | 19975 | -50.00 | 0.00 | 0.00 |
| 6 | T0001 | 122202 | 50.00 | 0.00 | 100.00 |
+----+-------+--------+---------+-------------+--------------+
UPDATE:
I just want the NUMBER values where they can be cancelled off. For T0001, it wouldn't matter which number it returned to cancel the negative value as long is it only returns one pair of values. For example, for T0001 it could return rows 4 and 5, 5 and 6, or 5 and 7. They would all be valid but I only want one of them.
This edit returns a single NUMBER for each CODE that matches your "zero-out" condition:
SELECT
CODE, MIN ( NUMBER ) AS MIN_NUMBER
FROM (
SELECT
ID,
CODE,
NUMBER,
BALANCE,
LAG ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LAG_BALANCE,
LEAD ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LEAD_BALANCE
FROM #Data
) AS Results
WHERE
BALANCE + LAG_BALANCE = 0
OR
BALANCE + LEAD_BALANCE = 0
GROUP BY
CODE
ORDER BY
CODE;
Returns
+-------+------------+
| CODE | MIN_NUMBER |
+-------+------------+
| B0001 | 122960 |
| T0001 | 19755 |
+-------+------------+
UPDATE #2:
/*
Return the first TWO rows for a CODE with BALANCEs that zero-out each other.
*/
SELECT
ID, CODE, NUMBER, BALANCE, ( BALANCE + LAG_BALANCE ) AS LAG_BALANCE, ( BALANCE + LEAD_BALANCE ) AS LEAD_BALANCE
FROM (
SELECT
ID,
CODE,
NUMBER,
BALANCE,
LAG ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LAG_BALANCE,
LEAD ( BALANCE, 1, 0 ) OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS LEAD_BALANCE,
ROW_NUMBER() OVER ( PARTITION BY CODE ORDER BY CODE, ID ) AS CODE_ROW
FROM #Data
) AS Results
WHERE
CODE_ROW <= 2
AND ( BALANCE + LAG_BALANCE = 0 OR BALANCE + LEAD_BALANCE = 0 )
ORDER BY
ID;
Returns
+----+-------+--------+---------+-------------+--------------+
| ID | CODE | NUMBER | BALANCE | LAG_BALANCE | LEAD_BALANCE |
+----+-------+--------+---------+-------------+--------------+
| 1 | B0001 | 122960 | 100.00 | 100.00 | 0.00 |
| 2 | B0001 | 123168 | -100.00 | 0.00 | 400.00 |
| 4 | T0001 | 19755 | 50.00 | 50.00 | 0.00 |
| 5 | T0001 | 19975 | -50.00 | 0.00 | 0.00 |
+----+-------+--------+---------+-------------+--------------+

You want to match rows on opposite balance, but each row should be matched only once.
An option is to enumerate the rows with row_number() first. You can then use the self-join solution, adding the row number in the join condition. I prefer not exists - but the logic is the same:
with cte as (
select
t.*,
row_number() over(partition by code, balance order by id) rn
from mytable t
)
select *
from cte c
where exists (
select 1
from cte c1
where c1.code = c.code and c1.rn = c.rn and c1.balance + c.balance = 0
)
order by code, id
Demo on DB Fiddle:
id | code | number | balance | rn
-: | :---- | -----: | ------: | -:
1 | B0001 | 122960 | 100.00 | 1
2 | B0001 | 123168 | -100.00 | 1
4 | T0001 | 19755 | 50.00 | 1
5 | T0001 | 19975 | -50.00 | 1

Something like this
;with
neg_cte as (select *, row_number() over(partition by code, balance order by id) rn
from #Data where BALANCE<0),
pos_cte as (select *, row_number() over(partition by code, balance order by id) rn
from #Data where BALANCE>0)
select * from neg_cte
union all
select pc.* from neg_cte nc join pos_cte pc on nc.CODE=pc.CODE
and nc.BALANCE=pc.BALANCE*-1
and nc.rn=pc.rn
order by ID;
Results
ID CODE NUMBER BALANCE rn
1 B0001 122960 100.00 1
2 B0001 123168 -100.00 1
4 T0001 19755 50.00 1
5 T0001 19975 -50.00 1

Related

Calculating Cost of Sales in SQL using FIFO method

From a stock transaction table I've created following sorted view table to be ready for FIFO:
rowN
date_
stockCode
sign_
amount
unitPrice
1
'2022-01-20'
ABC
in
5
29.20
2
'2022-01-22'
ABC
in
3
32.50
3
'2022-01-23'
ABC
out
7
40.00
4
'2022-01-23'
ABC
out
1
42.00
5
'2022-01-01'
XYZ
in
3
20.50
6
'2022-01-03'
XYZ
out
3
25.00
and I want to create a select query which looks like the previous table with only "out" rows and added cost_of_sales columns which is FIFO cost of that sale. But my knowledge of SQL is limited by just joins and sum over partitions.
The resulting table in my mind should look like this:
rowN
date_
stockCode
sign_
amount
unitPrice
cost_of_sales_uP
cost_of_sales
3
'2022-01-23'
ABC
out
7
40.00
30.1428
211.00
4
'2022-01-23'
ABC
out
1
42.00
32.50
32.50
6
'2022-01-03'
XYZ
out
3
25.00
20.50
61.50
I have no idea how to achieve this. Any help and guidance is appreciated. Result table doesn't have to be exactly like that but the main idea is there.
Thanks!
The following method might not be the fastest.
But it gets the job done.
First the incoming are unfolded into a temp table.
Then, by looping over the outgoing, each incoming unit is assigned to an outgoing on a First-In-First-Out basis.
The final query then uses the results of the temp table to calculate the total & average of the incoming.
IF OBJECT_ID('tempdb..#tmpStock') IS NOT NULL DROP TABLE #tmpStock;
CREATE TABLE #tmpStock (
id int identity primary key,
code varchar(30),
date_in date,
rowN int,
unitPrice decimal(10,2),
rowN_out int
);
--
-- Using a recursive CTE to unfold the incoming for the temp table
--
with RCTE as (
select stockCode, date_, rowN, amount, unitPrice
, 1 as lvl
from stock_transactions
where sign_ = 'in'
union all
select stockCode, date_, rowN, amount, unitPrice
, lvl + 1
from RCTE
where lvl < amount
)
insert into #tmpStock (code, date_in, rowN, unitPrice)
select stockCode, date_, rowN, unitPrice
from RCTE
order by stockCode, date_, rowN
option (maxrecursion 0);
DECLARE #IdOut INT = 1;
DECLARE #RowsOut INT = 0;
DECLARE #code VARCHAR(30);
DECLARE #amount SMALLINT;
DECLARE #date DATE;
DECLARE #rowN INT;
DECLARE #StockOut TABLE (
id int identity primary key,
code varchar(30),
date_out date,
rowN int,
amount smallint
);
insert into #StockOut (code, date_out, rowN, amount)
select stockCode, date_, rowN, amount
from stock_transactions
where sign_ = 'out'
order by stockCode, date_, rowN;
SELECT #RowsOut = COUNT(*) FROM #StockOut;
WHILE #IdOut <= #RowsOut
BEGIN
SELECT
#code = code
, #amount = amount
, #date = date_out
, #rowN = rowN
FROM #StockOut
WHERE id = #IdOut;
;WITH cte_in as (
select *
, rn = row_number() over (order by date_in, rowN)
from #tmpStock
where code = #code
and date_in <= #date
and rowN_out is null
)
UPDATE cte_in
SET rowN_out = #rowN
WHERE rn <= #amount;
SET #IdOut = #IdOut + 1;
END;
select * from #tmpStock
id | code | date_in | rowN | unitPrice | rowN_out
-: | :--- | :--------- | ---: | --------: | -------:
1 | ABC | 2022-01-20 | 1 | 29.20 | 3
2 | ABC | 2022-01-20 | 1 | 29.20 | 3
3 | ABC | 2022-01-20 | 1 | 29.20 | 3
4 | ABC | 2022-01-20 | 1 | 29.20 | 3
5 | ABC | 2022-01-20 | 1 | 29.20 | 3
6 | ABC | 2022-01-22 | 2 | 32.50 | 3
7 | ABC | 2022-01-22 | 2 | 32.50 | 3
8 | ABC | 2022-01-22 | 2 | 32.50 | 4
9 | XYZ | 2022-01-01 | 5 | 20.50 | 6
10 | XYZ | 2022-01-01 | 5 | 20.50 | 6
11 | XYZ | 2022-01-01 | 5 | 20.50 | 6
SELECT o.*
, CAST(i.AveragePriceIn AS DECIMAL(10,2)) AS cost_of_sales_uP
, i.TotalPriceIn AS cost_of_sales
FROM stock_transactions o
LEFT JOIN (
SELECT rowN_out
, AVG(unitPrice) as AveragePriceIn
, SUM(unitPrice) as TotalPriceIn
FROM #tmpStock
GROUP BY rowN_out
) i on i.rowN_out = o.rowN
WHERE o.sign_ = 'out'
ORDER BY o.rowN;
rowN | date_ | stockCode | sign_ | amount | unitPrice | cost_of_sales_uP | cost_of_sales
---: | :--------- | :-------- | :---- | -----: | --------: | ---------------: | ------------:
3 | 2022-01-23 | ABC | out | 7 | 40.00 | 30.14 | 211.00
4 | 2022-01-23 | ABC | out | 1 | 42.00 | 32.50 | 32.50
6 | 2022-01-03 | XYZ | out | 3 | 25.00 | 20.50 | 61.50
Demo on db<>fiddle here

Gaps And Islands: Splitting Islands Based On External Table

My scenario started off similar to a Island and Gaps problem, where I needed to find consecutive days of work. My current SQL query answers "ProductA was produced at LocationA from DateA through DateB, totaling X quantity".
However, this does not suffice when I needed to throw prices into the mix. Prices are in a separate table and handled in C# after the fact. Price changes are essentially a list of records that say "ProductA from LocationA is now Y value per unit effective DateC".
The end result is it works as long as the island does not overlap with a price-change date, but if it does overlap, I get a "close" answer, but it's not precise.
The C# code can handle applying the prices efficiently, what I need to do though is split the islands based on price changes. My goal is to make the SQL's partioning take into account the ranking of days from the other table, but I'm having trouble applying what I want to do.
The current SQL that generates my island is as follows
SELECT MIN(ScheduledDate) as StartDate, MAX(ScheduledDate) as
EndDate, ProductId, DestinationId, SUM(Quantity) as TotalQuantity
FROM (
SELECT ScheduledDate, DestinationId, ProductId, PartitionGroup = DATEADD(DAY ,-1 * DENSE_RANK() OVER (ORDER BY ScheduledDate), ScheduledDate), Quantity
FROM History
) tmp
GROUP BY PartitionGroup, DestinationId, ProductId;
The current SQL that takes from the PriceChange table and ranks the dates is as follows
DECLARE #PriceChangeDates TABLE(Rank int, SplitDate Date);
INSERT INTO #PriceChangeDates
SELECT DENSE_RANK() over (ORDER BY EffectiveDate) as Rank, EffectiveDate as SplitDate
FROM ProductPriceChange
GROUP BY EffectiveDate;
My thought is to somehow update the first queries inner SELECT statement to somehow take advantage of the #PriceChangeDates table created by the second query. I would think we can multiply the DATEADD's increment parameter by the rank from the declared table, but I am struggling to write it.
If I was to somehow do this with loops, my thought process would be to determine which rank the ScheduledDate would be from the #PriceChangeDates table, where its rank is the rank of the closest Date that is smaller than itself it can find. Then take whatever rank that gives and, I would think, multiply it by the increment parameter being passed in (or some math, for example doing a *#PriceChangeDates.Count() on the existing parameter and then adding in the new rank to avoid collisions). However, that's "loop" logic not "set" logic, and in SQL I need to think in sets.
Any and all help/advice is greatly appreciated. Thank you :)
UPDATE:
Sample data & example on SQLFiddle: http://www.sqlfiddle.com/#!18/af568/1
Where the data is:
CREATE TABLE History
(
ProductId int,
DestinationId int,
ScheduledDate date,
Quantity float
);
INSERT INTO History (ProductId, DestinationId, ScheduledDate, Quantity)
VALUES
(0, 1000, '20180401', 5),
(0, 1000, '20180402', 10),
(0, 1000, '20180403', 7),
(3, 5000, '20180507', 15),
(3, 5000, '20180508', 23),
(3, 5000, '20180509', 52),
(3, 5000, '20180510', 12),
(3, 5000, '20180511', 14);
CREATE TABLE PriceChange
(
ProductId int,
DestinationId int,
EffectiveDate date,
Price float
);
INSERT INTO PriceChange (ProductId, DestinationId, EffectiveDate, Price)
VALUES
(0, 1000, '20180201', 1),
(0, 1000, '20180402', 2),
(3, 5000, '20180101', 5),
(3, 5000, '20180510', 20);
The desired results would be to have a SQL statement that generates the result:
StartDate EndDate ProductId DestinationId TotalQuantity
2018-04-01 2018-04-01 0 1000 5
2018-04-02 2018-04-03 0 1000 17
2018-05-07 2018-05-09 3 5000 90
2018-05-10 2018-05-11 3 5000 26
To clarify, the end result does need the TotalQuantity of each split amount, so the procedural code that manipulates the results and applies the pricing knows how much of each product was one on each side of the price change to accurately determine the values.
Here is one more variant that is likely to perform better than my first answer. I decided to put it as a second answer, because the approach is rather different and the answer would be too long. You should compare performance of all variants with your real data on your hardware, and don't forget about indexes.
In the first variant I was using APPLY to pick a relevant price for each row in the History table. For each row from the History table the engine is searching for a relevant row from the PriceChange table. Even with appropriate index on the PriceChange table when this is done via a single seek, it still means 3.7 million seeks in a loop join.
We can simply join History and PriceChange tables together and with appropriate indexes on both tables it will be an efficient merge join.
Here I'm also using an extended sample data set to illustrate the gaps. I added these rows to the sample data from the question.
INSERT INTO History (ProductId, DestinationId, ScheduledDate, Quantity)
VALUES
(0, 1000, '20180601', 5),
(0, 1000, '20180602', 10),
(0, 1000, '20180603', 7),
(3, 5000, '20180607', 15),
(3, 5000, '20180608', 23),
(3, 5000, '20180609', 52),
(3, 5000, '20180610', 12),
(3, 5000, '20180611', 14);
Intermediate query
We do a FULL JOIN here, not a LEFT JOIN because it is possible that the date on which the price changed doesn't appear in the History table at all.
WITH
CTE_Join
AS
(
SELECT
ISNULL(History.ProductId, PriceChange.ProductID) AS ProductID
,ISNULL(History.DestinationId, PriceChange.DestinationId) AS DestinationId
,ISNULL(History.ScheduledDate, PriceChange.EffectiveDate) AS ScheduledDate
,History.Quantity
,PriceChange.Price
FROM
History
FULL JOIN PriceChange
ON PriceChange.ProductID = History.ProductID
AND PriceChange.DestinationId = History.DestinationId
AND PriceChange.EffectiveDate = History.ScheduledDate
)
,CTE2
AS
(
SELECT
ProductID
,DestinationId
,ScheduledDate
,Quantity
,Price
,MAX(CASE WHEN Price IS NOT NULL THEN ScheduledDate END)
OVER (PARTITION BY ProductID, DestinationId ORDER BY ScheduledDate
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS grp
FROM CTE_Join
)
SELECT *
FROM CTE2
ORDER BY
ProductID
,DestinationId
,ScheduledDate
Create the following indexes
CREATE UNIQUE NONCLUSTERED INDEX [IX_History] ON [dbo].[History]
(
[ProductId] ASC,
[DestinationId] ASC,
[ScheduledDate] ASC
)
INCLUDE ([Quantity])
CREATE UNIQUE NONCLUSTERED INDEX [IX_Price] ON [dbo].[PriceChange]
(
[ProductId] ASC,
[DestinationId] ASC,
[EffectiveDate] ASC
)
INCLUDE ([Price])
and the join will be an efficient MERGE join in the execution plan (not a LOOP join)
Intermediate result
+-----------+---------------+---------------+----------+-------+------------+
| ProductID | DestinationId | ScheduledDate | Quantity | Price | grp |
+-----------+---------------+---------------+----------+-------+------------+
| 0 | 1000 | 2018-02-01 | NULL | 1 | 2018-02-01 |
| 0 | 1000 | 2018-04-01 | 5 | NULL | 2018-02-01 |
| 0 | 1000 | 2018-04-02 | 10 | 2 | 2018-04-02 |
| 0 | 1000 | 2018-04-03 | 7 | NULL | 2018-04-02 |
| 0 | 1000 | 2018-06-01 | 5 | NULL | 2018-04-02 |
| 0 | 1000 | 2018-06-02 | 10 | NULL | 2018-04-02 |
| 0 | 1000 | 2018-06-03 | 7 | NULL | 2018-04-02 |
| 3 | 5000 | 2018-01-01 | NULL | 5 | 2018-01-01 |
| 3 | 5000 | 2018-05-07 | 15 | NULL | 2018-01-01 |
| 3 | 5000 | 2018-05-08 | 23 | NULL | 2018-01-01 |
| 3 | 5000 | 2018-05-09 | 52 | NULL | 2018-01-01 |
| 3 | 5000 | 2018-05-10 | 12 | 20 | 2018-05-10 |
| 3 | 5000 | 2018-05-11 | 14 | NULL | 2018-05-10 |
| 3 | 5000 | 2018-06-07 | 15 | NULL | 2018-05-10 |
| 3 | 5000 | 2018-06-08 | 23 | NULL | 2018-05-10 |
| 3 | 5000 | 2018-06-09 | 52 | NULL | 2018-05-10 |
| 3 | 5000 | 2018-06-10 | 12 | NULL | 2018-05-10 |
| 3 | 5000 | 2018-06-11 | 14 | NULL | 2018-05-10 |
+-----------+---------------+---------------+----------+-------+------------+
You can see that the Price column has a lot of NULL values. We need to "fill" these NULL values with the preceding non-NULL value.
Itzik Ben-Gan wrote a nice article showing how to solve this efficiently The Last non NULL Puzzle. Also see Best way to replace NULL with most recent non-null value.
This is done in CTE2 using MAX window function and you can see how it populates the grp column. This requires SQL Server 2012+. After the groups are determined we should remove rows where Quantity is NULL, because these rows are not from the History table.
Now we can do the same gaps-and-islands step using the grp column as an additional partitioning.
The rest of the query is pretty much the same as in the first variant.
Final query
WITH
CTE_Join
AS
(
SELECT
ISNULL(History.ProductId, PriceChange.ProductID) AS ProductID
,ISNULL(History.DestinationId, PriceChange.DestinationId) AS DestinationId
,ISNULL(History.ScheduledDate, PriceChange.EffectiveDate) AS ScheduledDate
,History.Quantity
,PriceChange.Price
FROM
History
FULL JOIN PriceChange
ON PriceChange.ProductID = History.ProductID
AND PriceChange.DestinationId = History.DestinationId
AND PriceChange.EffectiveDate = History.ScheduledDate
)
,CTE2
AS
(
SELECT
ProductID
,DestinationId
,ScheduledDate
,Quantity
,Price
,MAX(CASE WHEN Price IS NOT NULL THEN ScheduledDate END)
OVER (PARTITION BY ProductID, DestinationId ORDER BY ScheduledDate
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS grp
FROM CTE_Join
)
,CTE_RN
AS
(
SELECT
ProductID
,DestinationId
,ScheduledDate
,grp
,Quantity
,ROW_NUMBER() OVER (PARTITION BY ProductId, DestinationId, grp ORDER BY ScheduledDate) AS rn1
,DATEDIFF(day, '20000101', ScheduledDate) AS rn2
FROM CTE2
WHERE Quantity IS NOT NULL
)
SELECT
ProductId
,DestinationId
,MIN(ScheduledDate) AS StartDate
,MAX(ScheduledDate) AS EndDate
,SUM(Quantity) AS TotalQuantity
FROM
CTE_RN
GROUP BY
ProductId
,DestinationId
,grp
,rn2-rn1
ORDER BY
ProductID
,DestinationId
,StartDate
;
Final result
+-----------+---------------+------------+------------+---------------+
| ProductId | DestinationId | StartDate | EndDate | TotalQuantity |
+-----------+---------------+------------+------------+---------------+
| 0 | 1000 | 2018-04-01 | 2018-04-01 | 5 |
| 0 | 1000 | 2018-04-02 | 2018-04-03 | 17 |
| 0 | 1000 | 2018-06-01 | 2018-06-03 | 22 |
| 3 | 5000 | 2018-05-07 | 2018-05-09 | 90 |
| 3 | 5000 | 2018-05-10 | 2018-05-11 | 26 |
| 3 | 5000 | 2018-06-07 | 2018-06-11 | 116 |
+-----------+---------------+------------+------------+---------------+
This variant doesn't output the relevant price (as the first variant), because I simplified the "last non-null" query. It wasn't required in the question. In any case, it is pretty easy to add the price if needed.
The straight-forward method is to fetch the effective price for each row of History and then generate gaps and islands taking price into account.
It is not clear from the question what is the role of DestinationID. Sample data is of no help here.
I'll assume that we need to join and partition on both ProductID and DestinationID.
The following query returns effective Price for each row from History.
You need to add index to the PriceChange table
CREATE NONCLUSTERED INDEX [IX] ON [dbo].[PriceChange]
(
[ProductId] ASC,
[DestinationId] ASC,
[EffectiveDate] DESC
)
INCLUDE ([Price])
for this query to work efficiently.
Query for Prices
SELECT
History.ProductId
,History.DestinationId
,History.ScheduledDate
,History.Quantity
,A.Price
FROM
History
OUTER APPLY
(
SELECT TOP(1)
PriceChange.Price
FROM
PriceChange
WHERE
PriceChange.ProductID = History.ProductID
AND PriceChange.DestinationId = History.DestinationId
AND PriceChange.EffectiveDate <= History.ScheduledDate
ORDER BY
PriceChange.EffectiveDate DESC
) AS A
ORDER BY ProductID, ScheduledDate;
For each row from History there will be one seek in this index to pick the correct price.
This query returns:
Prices
+-----------+---------------+---------------+----------+-------+
| ProductId | DestinationId | ScheduledDate | Quantity | Price |
+-----------+---------------+---------------+----------+-------+
| 0 | 1000 | 2018-04-01 | 5 | 1 |
| 0 | 1000 | 2018-04-02 | 10 | 2 |
| 0 | 1000 | 2018-04-03 | 7 | 2 |
| 3 | 5000 | 2018-05-07 | 15 | 5 |
| 3 | 5000 | 2018-05-08 | 23 | 5 |
| 3 | 5000 | 2018-05-09 | 52 | 5 |
| 3 | 5000 | 2018-05-10 | 12 | 20 |
| 3 | 5000 | 2018-05-11 | 14 | 20 |
+-----------+---------------+---------------+----------+-------+
Now a standard gaps-and-island step to collapse consecutive days with the same price together. I use a difference of two row number sequences here.
I've added some more rows to your sample data to see the gaps within the same ProductId.
INSERT INTO History (ProductId, DestinationId, ScheduledDate, Quantity)
VALUES
(0, 1000, '20180601', 5),
(0, 1000, '20180602', 10),
(0, 1000, '20180603', 7),
(3, 5000, '20180607', 15),
(3, 5000, '20180608', 23),
(3, 5000, '20180609', 52),
(3, 5000, '20180610', 12),
(3, 5000, '20180611', 14);
If you run this intermediate query you'll see how it works:
WITH
CTE_Prices
AS
(
SELECT
History.ProductId
,History.DestinationId
,History.ScheduledDate
,History.Quantity
,A.Price
FROM
History
OUTER APPLY
(
SELECT TOP(1)
PriceChange.Price
FROM
PriceChange
WHERE
PriceChange.ProductID = History.ProductID
AND PriceChange.DestinationId = History.DestinationId
AND PriceChange.EffectiveDate <= History.ScheduledDate
ORDER BY
PriceChange.EffectiveDate DESC
) AS A
)
,CTE_rn
AS
(
SELECT
ProductId
,DestinationId
,ScheduledDate
,Quantity
,Price
,ROW_NUMBER() OVER (PARTITION BY ProductId, DestinationId, Price ORDER BY ScheduledDate) AS rn1
,DATEDIFF(day, '20000101', ScheduledDate) AS rn2
FROM
CTE_Prices
)
SELECT *
,rn2-rn1 AS Diff
FROM CTE_rn
Intermediate result
+-----------+---------------+---------------+----------+-------+-----+------+------+
| ProductId | DestinationId | ScheduledDate | Quantity | Price | rn1 | rn2 | Diff |
+-----------+---------------+---------------+----------+-------+-----+------+------+
| 0 | 1000 | 2018-04-01 | 5 | 1 | 1 | 6665 | 6664 |
| 0 | 1000 | 2018-04-02 | 10 | 2 | 1 | 6666 | 6665 |
| 0 | 1000 | 2018-04-03 | 7 | 2 | 2 | 6667 | 6665 |
| 0 | 1000 | 2018-06-01 | 5 | 2 | 3 | 6726 | 6723 |
| 0 | 1000 | 2018-06-02 | 10 | 2 | 4 | 6727 | 6723 |
| 0 | 1000 | 2018-06-03 | 7 | 2 | 5 | 6728 | 6723 |
| 3 | 5000 | 2018-05-07 | 15 | 5 | 1 | 6701 | 6700 |
| 3 | 5000 | 2018-05-08 | 23 | 5 | 2 | 6702 | 6700 |
| 3 | 5000 | 2018-05-09 | 52 | 5 | 3 | 6703 | 6700 |
| 3 | 5000 | 2018-05-10 | 12 | 20 | 1 | 6704 | 6703 |
| 3 | 5000 | 2018-05-11 | 14 | 20 | 2 | 6705 | 6703 |
| 3 | 5000 | 2018-06-07 | 15 | 20 | 3 | 6732 | 6729 |
| 3 | 5000 | 2018-06-08 | 23 | 20 | 4 | 6733 | 6729 |
| 3 | 5000 | 2018-06-09 | 52 | 20 | 5 | 6734 | 6729 |
| 3 | 5000 | 2018-06-10 | 12 | 20 | 6 | 6735 | 6729 |
| 3 | 5000 | 2018-06-11 | 14 | 20 | 7 | 6736 | 6729 |
+-----------+---------------+---------------+----------+-------+-----+------+------+
Now simply group by the Diff to get one row per interval.
Final query
WITH
CTE_Prices
AS
(
SELECT
History.ProductId
,History.DestinationId
,History.ScheduledDate
,History.Quantity
,A.Price
FROM
History
OUTER APPLY
(
SELECT TOP(1)
PriceChange.Price
FROM
PriceChange
WHERE
PriceChange.ProductID = History.ProductID
AND PriceChange.DestinationId = History.DestinationId
AND PriceChange.EffectiveDate <= History.ScheduledDate
ORDER BY
PriceChange.EffectiveDate DESC
) AS A
)
,CTE_rn
AS
(
SELECT
ProductId
,DestinationId
,ScheduledDate
,Quantity
,Price
,ROW_NUMBER() OVER (PARTITION BY ProductId, DestinationId, Price ORDER BY ScheduledDate) AS rn1
,DATEDIFF(day, '20000101', ScheduledDate) AS rn2
FROM
CTE_Prices
)
SELECT
ProductId
,DestinationId
,MIN(ScheduledDate) AS StartDate
,MAX(ScheduledDate) AS EndDate
,SUM(Quantity) AS TotalQuantity
,Price
FROM
CTE_rn
GROUP BY
ProductId
,DestinationId
,Price
,rn2-rn1
ORDER BY
ProductID
,DestinationId
,StartDate
;
Final result
+-----------+---------------+------------+------------+---------------+-------+
| ProductId | DestinationId | StartDate | EndDate | TotalQuantity | Price |
+-----------+---------------+------------+------------+---------------+-------+
| 0 | 1000 | 2018-04-01 | 2018-04-01 | 5 | 1 |
| 0 | 1000 | 2018-04-02 | 2018-04-03 | 17 | 2 |
| 0 | 1000 | 2018-06-01 | 2018-06-03 | 22 | 2 |
| 3 | 5000 | 2018-05-07 | 2018-05-09 | 90 | 5 |
| 3 | 5000 | 2018-05-10 | 2018-05-11 | 26 | 20 |
| 3 | 5000 | 2018-06-07 | 2018-06-11 | 116 | 20 |
+-----------+---------------+------------+------------+---------------+-------+
Not sure that i understand correctly, but this is just my idea:
Select concat_ws(',',view2.StartDate, string_agg(view1.splitDate, ','),
view2.EndDate), view2.productId, view2.DestinationId from (
SELECT DENSE_RANK() OVER (ORDER BY EffectiveDate) as Rank, EffectiveDate as
SplitDate FROM PriceChange GROUP BY EffectiveDate) view1 join
(
SELECT MIN(ScheduledDate) as StartDate, MAX(ScheduledDate) as
EndDate,ProductId, DestinationId, SUM(Quantity) as TotalQuantity
FROM (
SELECT ScheduledDate, DestinationId, ProductId, PartitionGroup =
DATEADD(DAY ,-1 * DENSE_RANK() OVER (ORDER BY ScheduledDate),
ScheduledDate), Quantity
FROM History
) tmp
GROUP BY PartitionGroup, DestinationId, ProductId
) view2 on view1.SplitDate >= view2.StartDate
and view1.SplitDate <=view2.EndDate
group by view2.startDate, view2.endDate, view2.productId,
view2.DestinationId
The result from this query will be:
| ranges | productId | DestinationId |
|---------------------------------------------|-----------|---------------|
| 2018-04-01,2018-04-02,2018-04-03 | 0 | 1000 |
| 2018-05-07,2018-05-10,2018-05-11 | 3 | 5000 |
Then, with any procedure language, for each row, you can split the string (with appropriate inclusive or exclusive rule for each boundary) to find out a list of condition (:from, :to, :productId, :destinationId).
And finally, you can loop through the list of conditions and use Union all clause to build one query (which is the union of all queries, which states a condition) to find out the final result. For example,
Select * from History where ScheduledDate >= '2018-04-01' and ScheduledDate <'2018-04-02' and productId = 0 and destinationId = 1000
union all
Select * from History where ScheduledDate >= '2018-04-02' and ScheduledDate <'2018-04-03' and productId = 0 and destinationId = 1000
----Update--------
Just based on above idea, i do some quick changes to provide your resultset. Maybe you can optimize it later
with view3 as
(Select concat_ws(',',view2.StartDate, string_agg(view1.splitDate, ','),
dateadd(day, 1, view2.EndDate)) dateRange, view2.productId, view2.DestinationId from (
SELECT DENSE_RANK() OVER (ORDER BY EffectiveDate) as Rank, EffectiveDate as
SplitDate FROM PriceChange GROUP BY EffectiveDate) view1 join
(
SELECT MIN(ScheduledDate) as StartDate, MAX(ScheduledDate) as
EndDate,ProductId, DestinationId, SUM(Quantity) as TotalQuantity
FROM (
SELECT ScheduledDate, DestinationId, ProductId, PartitionGroup =
DATEADD(DAY ,-1 * DENSE_RANK() OVER (ORDER BY ScheduledDate),
ScheduledDate), Quantity
FROM History
) tmp
GROUP BY PartitionGroup, DestinationId, ProductId
) view2 on view1.SplitDate >= view2.StartDate
and view1.SplitDate <=view2.EndDate
group by view2.startDate, view2.endDate, view2.productId,
view2.DestinationId
),
view4 as
(
select productId, destinationId, value from view3 cross apply string_split(dateRange, ',')
),
view5 as(
select *, row_number() over(partition by productId, destinationId order by value) rn from view4
),
view6 as (
select v52.value fr, v51.value t, v51.productid, v51. destinationid from view5 v51 join view5 v52
on v51.productid = v52.productid
and v51.destinationid = v52.destinationid
and v51.rn = v52.rn+1
)
select min(h.ScheduledDate) StartDate, max(h.ScheduledDate) EndDate, v6.productId, v6.destinationId, sum(h.quantity) TotalQuantity from view6 v6 join History h
on v6.destinationId = h.destinationId
and v6.productId = h.productId
and h.ScheduledDate >= v6.fr
and h.ScheduledDate <v6.t
group by v6.fr, v6.t, v6.productId, v6.destinationId
And the result is exactly the same with what you gave.
| StartDate | EndDate | productId | destinationId | TotalQuantity |
|------------|------------|-----------|---------------|---------------|
| 2018-04-01 | 2018-04-01 | 0 | 1000 | 5 |
| 2018-04-02 | 2018-04-03 | 0 | 1000 | 17 |
| 2018-05-07 | 2018-05-09 | 3 | 5000 | 90 |
| 2018-05-10 | 2018-05-11 | 3 | 5000 | 26 |
Use outer apply to choose the nearest price, then do a group by:
Live test: http://www.sqlfiddle.com/#!18/af568/65
select
StartDate = min(h.ScheduledDate),
EndDate = max(h.ScheduledDate),
h.ProductId,
h.DestinationId,
TotalQuantity = sum(h.Quantity)
from History h
outer apply
(
select top 1 pc.*
from PriceChange pc
where
pc.ProductId = h.ProductId
and pc.Effectivedate <= h.ScheduledDate
order by pc.EffectiveDate desc
) UpToDate
group by UpToDate.EffectiveDate,
h.ProductId,
h.DestinationId
order by StartDate, EndDate, ProductId
Output:
| StartDate | EndDate | ProductId | DestinationId | TotalQuantity |
|------------|------------|-----------|---------------|---------------|
| 2018-04-01 | 2018-04-01 | 0 | 1000 | 5 |
| 2018-04-02 | 2018-04-03 | 0 | 1000 | 17 |
| 2018-05-07 | 2018-05-09 | 3 | 5000 | 90 |
| 2018-05-10 | 2018-05-11 | 3 | 5000 | 26 |

Create calculated value based on calculated value inside previous row

I'm trying to find a way to apply monthly percentage changes to forecast pricing. I set my problem up in excel to make it a bit more clear. I'm using SQL Server 2017.
We'll say all months before 9/1/18 are historical and 9/1/18 and beyond are forecasts. I need to calculate the forecast price (shaded in yellow on the sample data) using...
Forecast Price = (Previous Row Forecast Price * Pct Change) + Previous Row Forecast Price
Just to be clear, the yellow shaded prices do not exist in my data yet. That is what I am trying to have my query calculate. Since this is monthly percentage change, each row depends on the row before and goes beyond a single ROW_NUMBER/PARTITION solution because we have to use the previous calculated price. Clearly what is an easy sequential calculation in excel is a bit more difficult here. Any idea how to create forecasted price column in SQL?
You need to use a recursive CTE. That is one of the easier ways to look at the value of a calculated value from previous row:
DECLARE #t TABLE(Date DATE, ID VARCHAR(10), Price DECIMAL(10, 2), PctChange DECIMAL(10, 2));
INSERT INTO #t VALUES
('2018-01-01', 'ABC', 100, NULL),
('2018-01-02', 'ABC', 150, 50.00),
('2018-01-03', 'ABC', 130, -13.33),
('2018-01-04', 'ABC', 120, -07.69),
('2018-01-05', 'ABC', 110, -08.33),
('2018-01-06', 'ABC', 120, 9.09),
('2018-01-07', 'ABC', 120, 0.00),
('2018-01-08', 'ABC', 100, -16.67),
('2018-01-09', 'ABC', NULL, -07.21),
('2018-01-10', 'ABC', NULL, 1.31),
('2018-01-11', 'ABC', NULL, 6.38),
('2018-01-12', 'ABC', NULL, -30.00),
('2019-01-01', 'ABC', NULL, 14.29),
('2019-01-02', 'ABC', NULL, 5.27);
WITH ncte AS (
-- number the rows sequentially without gaps
SELECT *, ROW_NUMBER() OVER (PARTITION BY ID ORDER BY Date) AS rn
FROM #t
), rcte AS (
-- find first row in each group
SELECT *, Price AS ForecastedPrice
FROM ncte AS base
WHERE rn = 1
UNION ALL
-- find next row for each group from prev rows
SELECT curr.*, CAST(prev.ForecastedPrice * (1 + curr.PctChange / 100) AS DECIMAL(10, 2))
FROM ncte AS curr
INNER JOIN rcte AS prev ON curr.ID = prev.ID AND curr.rn = prev.rn + 1
)
SELECT *
FROM rcte
ORDER BY ID, rn
Result:
| Date | ID | Price | PctChange | rn | ForecastedPrice |
|------------|-----|--------|-----------|----|-----------------|
| 2018-01-01 | ABC | 100.00 | NULL | 1 | 100.00 |
| 2018-01-02 | ABC | 150.00 | 50.00 | 2 | 150.00 |
| 2018-01-03 | ABC | 130.00 | -13.33 | 3 | 130.01 |
| 2018-01-04 | ABC | 120.00 | -7.69 | 4 | 120.01 |
| 2018-01-05 | ABC | 110.00 | -8.33 | 5 | 110.01 |
| 2018-01-06 | ABC | 120.00 | 9.09 | 6 | 120.01 |
| 2018-01-07 | ABC | 120.00 | 0.00 | 7 | 120.01 |
| 2018-01-08 | ABC | 100.00 | -16.67 | 8 | 100.00 |
| 2018-01-09 | ABC | NULL | -7.21 | 9 | 92.79 |
| 2018-01-10 | ABC | NULL | 1.31 | 10 | 94.01 |
| 2018-01-11 | ABC | NULL | 6.38 | 11 | 100.01 |
| 2018-01-12 | ABC | NULL | -30.00 | 12 | 70.01 |
| 2019-01-01 | ABC | NULL | 14.29 | 13 | 80.01 |
| 2019-01-02 | ABC | NULL | 5.27 | 14 | 84.23 |
Demo on DB Fiddle
In SQL Server you can access values of previous / next rows by using the windowing functions LAG and LEAD. You need to define the order of the rows by specifying it in the OVER clause. You may need to wrap the select query, that returns prev/next values in a derived table or CTE, and then select from it and calculate your forecasts.
with cte as (SELECT [Date], Price, LAG(Price, 1) over(order by [Date]) as PrevPrice from TABLE)
select [Date], Price, Price - PrevPrice as PriceChange from cte

Add extra column in sql to show ratio with previous row

I have a SQL table with a format like this:
SELECT period_id, amount FROM table;
+--------------------+
| period_id | amount |
+-----------+--------+
| 1 | 12 |
| 2 | 11 |
| 3 | 15 |
| 4 | 20 |
| .. | .. |
+-----------+--------+
I'd like to add an extra column (just in my select statement) that calculates the growth ratio with the previous amount, like so:
SELECT period_id, amount, [insert formula here] AS growth FROM table;
+-----------------------------+
| period_id | amount | growth |
+-----------+-----------------+
| 1 | 12 | |
| 2 | 11 | 0.91 | <-- 11/12
| 3 | 15 | 1.36 | <-- 15/11
| 4 | 20 | 1.33 | <-- 20/15
| .. | .. | .. |
+-----------+-----------------+
Just need to work out how to perform the operation with the line before. Not interested in adding to the table. Any help appreciated :)
** also want to point out that period_id is in order but not necessarily increasing incrementally
The window function Lag() would be a good fit here.
You may notice that we use (amount+0.0). This is done just in case AMOUNT is an INT, and NullIf() to avoid the dreaded divide by zero
Declare #YourTable table (period_id int,amount int)
Insert Into #YourTable values
( 1,12),
( 2,11),
( 3,15),
( 4,20)
Select period_id
,amount
,growth = cast((amount+0.0) / NullIf(lag(amount,1) over (Order By Period_ID),0) as decimal(10,2))
From #YourTable
Returns
period_id amount growth
1 12 NULL
2 11 0.92
3 15 1.36
4 20 1.33
If you are using SQL Server 2012+ then go for John Cappelletti answer.
And If you are also less blessed like me then this below code work for you in the 2008 version too.
Declare #YourTable table (period_id int,amount int)
Insert Into #YourTable values
( 1,12),
( 2,11),
( 3,15),
( 4,20)
;WITH CTE AS (
SELECT ROW_NUMBER() OVER (
ORDER BY period_id
) SNO
,period_id
,amount
FROM #YourTable
)
SELECT C1.period_id
,C1.amount
,CASE
WHEN C2.amount IS NOT NULL AND C2.amount<>0
THEN CAST(C1.amount / CAST(C2.amount AS FLOAT) AS DECIMAL(18, 2))
END AS growth
FROM CTE C1
LEFT JOIN CTE C2 ON C1.SNO = C2.SNO + 1
Which works same as LAG.
+-----------+--------+--------+
| period_id | amount | growth |
+-----------+--------+--------+
| 1 | 12 | NULL |
| 2 | 11 | 0.92 |
| 3 | 15 | 1.36 |
| 4 | 20 | 1.33 |
+-----------+--------+--------+

Top Dense_Rank row based on other fields

I have several tables tied together in sql that I am trying to display only the MAX number from a column formulated using DENSE RANK but I need to keep in mind 2 other fields when pulling the TOP row.
Here is a sample of my result:
| sa_id | price | threshold | role_id | rk
1 | 37E41 | 40.00 | NULL | A38D67A | 1
2 | 37E41 | 40.00 | NULL | 46B9D4E | 1
3 | 1CFC1 | 40.00 | NULL | 58C1E03 | 1
4 | BF0D3 | 40.00 | NULL | 28D465B | 1
5 | F914B | 40.00 | NULL | 2920EBD | 1
6 | F3CA1 | 40.00 | NULL | D5E7584 | 1
7 | 0D8C1 | 40.00 | NULL | EECDB5A | 1
8 | A6503 | 40.00 | NULL | B680CB4 | 1
9 | 9BB96 | 40.00 | 0.01 | D66E612 | 1
10 | 9BB96 | 40.00 | 20.03 | D66E612 | 2
11 | 9BB96 | 40.00 | 40.03 | D66E612 | 3
12 | 9BB96 | 40.00 | 60.03 | D66E612 | 4
13 | 9BB96 | 40.00 | 80.03 | D66E612 | 5
What I am hoping to accomplish is to display all columns in this screenshot using the highest value for rk (calculated using DENSE RANK) where price > threshold and the sa_id & role_id are unique.
In this case I would want to display the following rows only: 1, 2, 3, 4, 5, 6, 7, 8, 10
Is this possible?
SELECT
servicerate_audit_id as sa_id
,ticket_price as price
,threshold_threshold/100.00 as threshold
,charge_role.chargerole_id as role_id
,DENSE_RANK() OVER(
PARTITION BY threshold_audit_id
ORDER BY
ISNULL(threshold_threshold,9999999),
threshold_threshold
) as rk
FROM sts_service_charge_rate
INNER JOIN ts_threshold
ON threshold_id = servicerate_threshold_id
INNER JOIN ts_charge_role as charge_role
ON chargerole_id = servicerate_charge_role_id
If you can modify your original query:
SELECT *
FROM (
SELECT
servicerate_audit_id as sa_id
,ticket_price as price
,threshold_threshold/100.00 as threshold
,charge_role.chargerole_id as role_id
,DENSE_RANK() OVER(
PARTITION BY threshold_audit_id
ORDER BY
ISNULL(threshold_threshold,9999999),
threshold_threshold
) as rk
,DENSE_RANK() OVER(
ORDER BY
ISNULL(threshold_threshold,9999999) DESC,
threshold_threshold DESC
) as rk_inverse
FROM sts_service_charge_rate
INNER JOIN ts_threshold
ON threshold_id = servicerate_threshold_id
INNER JOIN ts_charge_role as charge_role
ON chargerole_id = servicerate_charge_role_id
) t
WHERE price > COALESCE(threshold, 0)
AND t.rk_inverse = 1
Observe I just added an inverse calculation of your ranking and filtered for the top rk_inverse per partition. I'm assuming that the PARTITION BY threshold_audit_id and your requirement of having unique (sa_id, role_id) tuples are functionally dependent. Otherwise, your rk_inverse calculation would need to take into consideration a different PARTITION BY clause.
If you cannot modify your original query:
You can calculate another window function that orders your rk values descendingly (highest first) per your partition (sa_id, role_id), and then take only the top one per partition:
SELECT sa_id, price, threshold, role_id, rk
FROM (
SELECT result.*, row_number() OVER (PARTITION BY sa_id, role_id ORDER BY rk DESC) rn
FROM (... original query ...)
WHERE price > COALESCE(threshold, 0)
) t
WHERE rn = 1