I have a SQL table that I need to count the rows with 0 turnover, but the challenge is they resets. I only need the number of consecutive rows since it last generated any turnover.
Source data (it has a lot of different ID, just using 442 and 4500 in this case):
ID |Date | T/O |
442 |2019-12-31 | 0 |
442 |2020-01-01 |200.00|
442 |2020-01-02 | 0 |
442 |2020-02-06 | 0 |
442 |2020-02-07 | 0 |
442 |2020-02-08 | 0 |
442 |2020-02-09 |150.00|
442 |2020-02-10 | 0 |
442 |2020-02-11 | 0 |
442 |2020-02-15 | 0 |
4500 |2020-01-01 | 0 |
Intended results:
442 | 3 |
4500 | 1 |
I thought of using LAG(), but the number of rows between turnover generated can vary significantly. Sometimes it can be even 30+ rows.
SELECT id, COUNT(*) as [result]
FROM SourceData sd1
WHERE t_o=0
AND NOT EXISTS (SELECT 1
FROM SourceData sd2
WHERE sd1.id=sd2.id AND t_o != 0 AND sd2.[Date] > sd1.[Date])
GROUP BY id
DEMO
First we can get the last non-zero date for each id.
select id, max(date) as date
from example
where t_o > 0
group by id
This will not show a value for 4500 because it lacks a non-zero value.
Then we can use this to select and group only the values after those dates, or all rows if there was no non-zero date for an id.
with last_to as(
select id, max(date) as date
from example
where t_o > 0
group by id
)
select example.id, count(example.t_o)
from example
-- Use a left join to get all ids in example,
-- even those missing from last_to.
left join last_to on last_to.id = example.id
-- Account for the lack of a last_to row
-- if the ID has no non-zero values.
where last_to.date is null
or example.date > last_to.date
group by example.id
Demonstration.
Background
I need to implement inventory valuation / costing using the FIFO (first-in, first-out) method.
I'm running Postgres 11 running on CentOS 7.
I've looked at, and tried, a fair number of hypotheses from SO and the wider internet (as well as searching my own print library which includes SQL Queries for Mere Mortals, PostgreSQL Up & Running, The SQL Cookbook, Practical Issues In Database Management, and other quality reference works), and to date, I can't find a solution that works for closing inventory valuation.
(I've also tried reasoning it out on my own, but have failed to come up with a plausible appraoch)
NOTE In my case, I have permission to change the table structure, etc, of the setup, so I can add / remove / change anything in the setup as needed (such as, e.g., adding a direction column to the movements table, as some approaches I've tried have indicated, changing queries, etc etc)
Current setup
I have a table mockup_inv_movements:
CREATE TABLE the_schema.mockup_inv_movements (
id INTEGER NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
created_at TIMESTAMP WITH TIME ZONE DEFAULT now(),
sku TEXT,
adjustment_quantity NUMERIC,
unit_cost NUMERIC(19,2),
po_num INTEGER
);
and this view mockup_inv_movements_with_fifo_cost adds FIFO cost for sale / 'out' rows, calculated from a query (shown later below):
CREATE VIEW the_schema.mockup_inv_movements_with_fifo_cost AS (
select
i.id,
i.created_at,
i.po_num,
i.sku,
i.adjustment_quantity,
i.unit_cost,
m.fifo_unit_cost
FROM
the_schema.mockup_inv_movements i
LEFT OUTER JOIN
the_schema.fifo_hypothesis_2_mockup m
ON
i.id = m.id
ORDER BY i.id
);
Adding some test inventory movement data:
-- insert receipt / 'in' records
INSERT INTO the_schema.mockup_inv_movements (sku, adjustment_quantity, unit_cost, po_num, created_at )
VALUES ('foo_product',100,4,123, now()+'1 hour'), ('foo_product',10,3,987, now()+'2 hour'), ('foo_product',20,7,223, now()+'3 hours')
;
INSERT INTO the_schema.mockup_inv_movements (sku, adjustment_quantity, unit_cost, po_num, created_at )
VALUES ('bar_product',100,5,123, now()+'4 hours'),('bar_product',30,6,963, now()+'5 hours'),('bar_product',50,8,223, now()+'6 hours'),('bar_product',5,5,456, now()+'7 hours')
;
--insert sale / 'out' records
INSERT INTO the_schema.mockup_inv_movements (sku, adjustment_quantity, unit_cost, po_num, created_at )
VALUES ('bar_product',-50,null,null, now()+'8 hours'),('bar_product',-30, null,null, now()+'9 hours'),
('bar_product',-20,null,null, now()+'10 hours'),('bar_product',-10,null,null, now()+'11 hours')
;
INSERT INTO the_schema.mockup_inv_movements (sku, adjustment_quantity, unit_cost, po_num, created_at )
VALUES ('foo_product',-70,null,null, now()+'12 hours'), ('foo_product',-5,null,null, now()+'13 hours'),
('foo_product',-20,null,null, now()+'14 hours'),('foo_product',-10,null,null, now()+'15 hours')
;
OK, now here's the query that calculates the 'sale/out' price for each, taken from this question, which seems to work; note that I'm only pulling in the column fifo_unit_cost from this query at the moment:
CREATE VIEW the_schema.fifo_hypothesis_2_mockup AS (
SELECT
id,
sku,
created_at AT TIME ZONE 'mst',
qty_sold,
-- 5
round((cumulative_sold_cost - coalesce(lag(cumulative_sold_cost) over w, 0))/qty_sold, 2) as fifo_unit_cost,
qty_bought,
prev_bought,
total_cost,
prev_total_cost,
cumulative_sold_cost,
coalesce(lag(cumulative_sold_cost) over w, 0) as prev_cumulative_sold_cost
FROM (
SELECT id,
tneg.sku,
created_at,
qty_sold,
tpos.qty_bought,
prev_bought,
total_cost,
prev_total_cost,
-- 4
round(prev_total_cost + ((tneg.cumulative_sold - tpos.prev_bought)/(tpos.qty_bought - tpos.prev_bought))*(total_cost-prev_total_cost), 2) as cumulative_sold_cost
FROM (
SELECT
id,
sku,
created_at,
-(adjustment_quantity) as qty_sold,
sum(-(adjustment_quantity)) over w as cumulative_sold
FROM the_schema.mockup_inv_movements
WHERE adjustment_quantity < 0
WINDOW w AS (PARTITION BY sku ORDER BY created_at)
-- 1
) tneg
LEFT JOIN (
SELECT
sku,
sum(adjustment_quantity) over w as qty_bought,
coalesce(sum(adjustment_quantity) over prevw, 0) as prev_bought,
adjustment_quantity * unit_cost as cost,
sum(adjustment_quantity * unit_cost) over w as total_cost,
coalesce(sum(adjustment_quantity * unit_cost) over prevw, 0) as prev_total_cost
FROM the_schema.mockup_inv_movements
WHERE adjustment_quantity > 0
WINDOW w AS (PARTITION BY sku ORDER BY created_at),
prevw AS (PARTITION BY sku ORDER BY created_at ROWS BETWEEN unbounded preceding AND 1 preceding)
-- 2
) tpos
-- 3
ON
((tneg.cumulative_sold > tpos.prev_bought )
AND ( tneg.cumulative_sold <= tpos.qty_bought ))
AND tneg.sku = tpos.sku
) t
WINDOW w AS (PARTITION BY sku ORDER BY created_at)
ORDER BY id
)
;
Now here's the part where I'm having trouble.
I need to calculate the value of remaining stock / inventory on hand, also known as "closing stock" or "closing inventory." I've tried a number of approaches including this question and this 'set-based speed phreakery' method, the latter of which I readily admit that I don't fully comprehend,
The approach that has come closest to working for me is this older hypothesis from Ranjeet Rana, BUT although it does seem to assign the FIFO costs according to the correct breakdown, the sum of closing stock for each SKU does not seem to match the raw difference between 'in' and 'out' quantities.
Here's the closing stock query adapted from Rana (comments mine; I left them in just in case they might indicate where my error is).
CREATE VIEW the_schema.closing_inv_hyp_3 AS (
select *,
case
when cumulative>0 and adjustment_quantity>=cumulative -- note that sale/out adjustment_quantity / cumulative is always less than zero
then cumulative*cost -- in this case, some amount of this row's receipt has been sold, and the remainder qty is shown in 'cumulative'
when cumulative>0 and adjustment_quantity<cumulative
then adjustment_quantity*cost -- in this case, none of this row's receipt has been sold, and so the entire adjustment amount is multiplied by unit cost
else 0 -- sale rows are assigned zero for this column
end as closing_stock
from (
select
*, -- all rows from subquery
sum(adjustment_quantity) over (order by srl) as cumulative -- THIS is the problematic column
from (
select
0 as srl, -- this ensures that all 'sale / out' rows float to the top
id,
sku,
adjustment_quantity,
COALESCE(fifo_unit_cost,unit_cost) AS cost,
created_at
from
the_schema.mockup_inv_movements_with_fifo_cost
where adjustment_quantity < 0 -- SALE / OUT only
UNION -- gets all from both queries (less any dupes)
select
row_number() over(order by created_at) as srl, -- this assigns a synthetic sequential row number to the 'PO / in' rows and ensures the are pushed to the bottom
id,
sku,
adjustment_quantity,
COALESCE(fifo_unit_cost,unit_cost) AS cost,
created_at
from
the_schema.mockup_inv_movements_with_fifo_cost
where
adjustment_quantity > 0 -- PO / IN only
ORDER BY srl
)as tab
) as maintab
);
With this in place, we should be able to get the sum of closing stock value per SKU with:
SELECT
sku,
sum(closing_stock) as closing_stock_sum_value
FROM
the_schema.closing_inv_hyp_3
WHERE closing_stock > 0
GROUP BY sku
ORDER by sku
;
However, as I mentioned, the totals do not match up with the basic inventory difference calculation (specifically in this test example, I would expect 75 units of bar_product to be represented in closing stock, whereas this query shows 100):
srl | id | sku | adjustment_quantity | cost | created_at | cumulative | closing_stock
-----+-----+-------------+---------------------+------+-------------------------------+------------+---------------
0 | 102 | foo_product | -70 | 4.00 | 2022-03-10 07:27:05.447572+00 | -215 | 0
0 | 100 | bar_product | -20 | 5.00 | 2022-03-10 05:27:05.447572+00 | -215 | 0
0 | 101 | bar_product | -10 | 6.00 | 2022-03-10 06:27:05.447572+00 | -215 | 0
0 | 103 | foo_product | -5 | 4.00 | 2022-03-10 08:27:05.447572+00 | -215 | 0
0 | 105 | foo_product | -10 | 3.50 | 2022-03-10 10:27:05.447572+00 | -215 | 0
0 | 99 | bar_product | -30 | 5.00 | 2022-03-10 04:27:05.447572+00 | -215 | 0
0 | 98 | bar_product | -50 | 5.00 | 2022-03-10 03:27:05.447572+00 | -215 | 0
0 | 104 | foo_product | -20 | 4.00 | 2022-03-10 09:27:05.447572+00 | -215 | 0
1 | 91 | foo_product | 100 | 4.00 | 2022-03-09 20:27:05.447572+00 | -115 | 0
2 | 92 | foo_product | 10 | 3.00 | 2022-03-09 21:27:05.447572+00 | -105 | 0
3 | 93 | foo_product | 20 | 7.00 | 2022-03-09 22:27:05.447572+00 | -85 | 0
4 | 94 | bar_product | 100 | 5.00 | 2022-03-09 23:27:05.447572+00 | 15 | 75.00
5 | 95 | bar_product | 30 | 6.00 | 2022-03-10 00:27:05.447572+00 | 45 | 180.00
6 | 96 | bar_product | 50 | 8.00 | 2022-03-10 01:27:05.447572+00 | 95 | 400.00
7 | 97 | bar_product | 5 | 5.00 | 2022-03-10 02:27:05.447572+00 | 100 | 25.00
(15 rows)
It seems like this would be the kind of thing that has a more-or-less standardized solution, but so far none of the resources I've found / tried has guided me to a working approach.
How can I accurately do FIFO closing stock / inventory valuation in Postgres?
All guidance much appreciated!
Using "Set-based Speed Phreakery: The FIFO Stock Inventory SQL Problem" as an example, re-working that approach for Postgres and the change of table/columns produces this query:
/* Sum up the ins and outs to calculate the remaining stock level */
WITH cteStockSum
AS ( SELECT sku ,
SUM(adjustment_quantity) AS TotalStock
FROM mockup_inv_movements
GROUP BY sku
)
, cteReverseInSum
AS ( SELECT s.sku ,
s.created_at ,
( SELECT SUM(i.adjustment_quantity)
FROM mockup_inv_movements AS i
WHERE i.sku = s.sku
AND i.adjustment_quantity > 0
AND i.created_at >= s.created_at
) AS RollingStock ,
s.adjustment_quantity AS ThisStock
FROM mockup_inv_movements AS s
WHERE s.adjustment_quantity > 0
)
/* Using the rolling balance above find the first stock movement in that meets
(or exceeds) our required stock level */
/* and calculate how much stock is required from the earliest stock in */
, cteWithLastTranDate
AS ( SELECT w.sku ,
w.TotalStock ,
LastPartialStock.created_at ,
LastPartialStock.StockToUse ,
LastPartialStock.RunningTotal ,
w.TotalStock - LastPartialStock.RunningTotal
+ LastPartialStock.StockToUse AS UseThisStock
FROM cteStockSum AS w
CROSS JOIN LATERAL ( SELECT
z.created_at ,
z.ThisStock AS StockToUse ,
z.RollingStock AS RunningTotal
FROM cteReverseInSum AS z
WHERE z.sku = w.sku
AND z.RollingStock >= w.TotalStock
ORDER BY z.created_at DESC
LIMIT 1
) AS LastPartialStock
)
/* Sum up the cost of 100% of the stock movements in after the returned stockid and for that stockid we need 'UseThisStock' items' */
SELECT y.sku ,
y.TotalStock AS CurrentItems ,
SUM(CASE WHEN e.created_at = y.created_at THEN y.UseThisStock
ELSE e.adjustment_quantity
END * Price.unit_cost) AS CurrentValue
FROM cteWithLastTranDate AS y
INNER JOIN mockup_inv_movements AS e
ON e.SKU = y.SKU
AND e.created_at >= y.created_at
AND e.adjustment_quantity > 0
CROSS JOIN LATERAL (
/* Find the Price of the item in */ SELECT
p.unit_cost
FROM mockup_inv_movements AS p
WHERE p.SKU = e.SKU
AND p.created_at <= e.created_at
AND p.adjustment_quantity > 0
ORDER BY p.created_at DESC
LIMIT 1
) AS Price
GROUP BY y.sku ,y.TotalStock
ORDER BY y.sku
and from your sample data the result produced is this:
+-------------+--------------+--------------+
| sku | currentitems | currentvalue |
+-------------+--------------+--------------+
| bar_product | 75 | 545.00 |
| foo_product | 25 | 155.00 |
+-------------+--------------+--------------+
also see: https://dbfiddle.uk/?rdbms=postgres_11&fiddle=f564a6cfda3374c2057b437f845a4bdf
Here's my dataset of trades, traders and counterparties:
TRADER_ID | TRADER_NAME | EXEC_BROKER | TRADE_AMOUNT | TRADE_ID
ABC123 | Jules Winnfield | GOLD | 10000 | ASDADAD
XDA241 | Jimmie Dimmick | GOLD | 12000 | ASSVASD
ADC123 | Vincent Vega | BARC | 10000 | ZXCZCX
ABC123 | Jules Winnfield | BARC | 15000 | ASSXCQA
ADC123 | Vincent Vega | CRED | 250000 | RFAQQA
ABC123 | Jules Winnfield | CRED | 5000 | ASDQ23A
ABC123 | Jules Winnfield | GOLD | 5000 | AVBDQ3A
I'm looking to produce a repeatable monthly report that gives me a view of trading activity aggregated at the counterparty (the EXEC_BROKER field) level, with subtotals - as shown below:
TRADER_ID | TRADER_NAME | NO._OF_CCP_USED | CCP | TRADED_AMT_WITH_CCP | VALUE_OF_TOTAL_TRADES | TRADES_WITH_CCP | TOTAL_TRADES
ABC123 | Jules Winnfield | 3 | GOLD | 15000 | 35000 | 2 | 4
ABC123 | Jules Winnfield | 3 | BARC | 15000 | 35000 | 1 | 4
ABC123 | Jules Winnfield | 3 | CRED | 5000 | 35000 | 1 | 4
...and so on the rest.
The idea is to aggregate the number of trades per counterparty (which I have done using a count function), and the sum of traded amounts with the ccp, but I'm struggling to get the 'subtotal' field next to each trader as shown in my desired output above - so you can see here that Jules has dealt with 3 counterparties in total, with 4 trades between them, and a collective amount of 35000.
I have tried using a combination of aggregate and over by functions, but to no avail.
SELECT
OT.TRADER_ID,
OT.TRADER_NAME,
OT.EXEC_BROKER,
SUM(OT.TRADE_AMOUNT) AS VALUE_OF_TOTAL_TRADES,
COUNT(OT.TRADE_ID) AS TOTAL_TRADES,
COUNT(OT.EXEC_BROKER) OVER PARTITION BY (OT.TRADER_ID) AS NO._OF_CCP_USED,
SUM(OT.TRADE_AMOUNT) OVER PARTITION BY (OT.EXEC_BROKER) AS TRADED_AMT_WITH_CCP,
COUNT(OT.TRADE_ID) OVER PARTITION BY (OT.EXEC_BROKER) AS TRADES_WITH_CCP
FROM dbo.ORDERS_TRADES OT
GROUP BY OT.TRADER_ID, OT.TRADER_NAME, OT.EXEC_BROKER, OT.TRADE_AMOUNT, OT.TRADE_ID
The code above runs but returns millions of rows. When I remove the partition by lines, I get the desired result minus the subtotal columns I'm looking for.
Any suggestions please? Thanks very much!
EDIT:
Final code which gave me the desired output: updating my question to provide this response (thanks to Gordon Linoff) so that others can benefit:
SELECT
OT.TRADER_ID,
OT.TRADER_NAME,
OT.EXEC_BROKER,
RANK() OVER (PARTITION BY OT.TRADER_ID ORDER BY
SUM(OT.TRADE_AMOUNT) DESC) AS CCP_RANK,
SUM(OT.TRADE_AMOUNT) AS TRADED_AMT_WITH_CCP,
SUM(SUM(OT.TRADE_AMOUNT)) OVER (PARTITION BY OT.TRADER_ID) AS
VALUE_OF_TOTAL_TRADES,
COUNT(*) OVER (PARTITION BY OT.TRADER_ID) AS NUM_OF_CCP_USED,
SUM(COUNT(OT.TRADE_ID)) OVER (PARTITION BY OT.TRADER_ID) AS
TOTAL_TRADES
FROM dbo.ORDERS_TRADES OT
GROUP BY OT.TRADER_ID, OT.TRADER_NAME, OT.EXEC_BROKER
You seem to want:
SELECT OT.TRADER_ID, OT.TRADER_NAME, OT.CCP,
COUNT(*) OVER (PARTITION BY OT.TRADER_ID) as NUM_CCP,
SUM(OT.TRADED_AMT) AS TRADED_AMT_WITH_CCP,
SUM(SUM(OT.TRADED_AMT)) OVER (PARTITION BY OT.TRADER_ID) AS VALUE_OF_TOTAL_TRADES,
COUNT(OT.TRADE_ID) AS CCP_TRADES,
SUM(COUNT(OT.TRADE_ID)) OVER (PARTITION BY OT.TRADER_ID) AS TOTAL_TRADES
FROM ORDERS_TRADES OT
GROUP BY OT.TRADER_ID, OT.TRADER_NAME, OT.CCP;
I'm not sure what your query has to do with the results you want. The columns have little to do with what you are asking.
Here is a db<>fiddle.
Making some assumptions about the nomenclature, here is a solution that doesn't use anything too fancy so it's easy to maintain, though it's not the most efficient:
create table trades
(
TRADER_ID varchar(10),
TRADER_NAME varchar(20),
CCP char(4),
TRADED_AMT decimal(10,2),
TRADE_ID varchar(10) primary key
);
insert trades
values
('ABC123', 'Jules Winnfield', 'GOLD', 10000 , 'ASDADAD'),
('XDA241', 'Jimmie Dimmick ', 'GOLD', 12000 , 'ASSVASD'),
('ADC123', 'Vincent Vega ', 'BARC', 10000 , 'ZXCZCX'),
('ABC123', 'Jules Winnfield', 'BARC', 15000 , 'ASSXCQA'),
('ADC123', 'Vincent Vega ', 'CRED', 250000, 'RFAQQA'),
('ABC123', 'Jules Winnfield', 'CRED', 5000 , 'ASDQ23A'),
('ABC123', 'Jules Winnfield', 'GOLD', 5000 , 'AVBDQ3A');
with trader_totals as
(
select trader_id,
distinct_ccps = count(distinct CCP),
total_amt = sum(traded_amt),
total_count = count(*)
from trades
group by trader_id
)
select trader_id = tr.trader_id,
trader_name = trader_name,
distinct_CCP_count = tt.distinct_ccps,
CCP = tr.CCP,
this_CCP_traded_amt = sum(traded_amt),
total_traded_amt = tt.total_amt,
this_CCP_traded_count = count(*),
total_traded_count = tt.total_count
from trades tr
join trader_totals tt on tt.trader_id = tr.trader_id
group by tr.trader_id,
tr.trader_name,
tr.CCP,
tt.distinct_ccps,
tt.total_amt,
tt.total_count
I'm trying to sum some values from one column (total_stake) based on the second column (node_id) and group results by node_id. Right now it sums everything perfectly but it's still duplicate rows with the same node_id and summed value and I don't fully understand why.
Here is my query:
WITH events AS (
SELECT n.id as node_id, n.event_time FROM nodes n
)
SELECT
node_id,
sum(total) FILTER (WHERE prior_to=0 OR prior_to=2) OVER (PARTITION BY node_id) as node_total_previous_days,
sum(total) FILTER (WHERE prior_to=1) OVER (PARTITION BY node_id) as node_total_same_day,
sum(total) FILTER (WHERE prior_to=2) OVER (PARTITION BY node_id) as node_total_previous_day,
FROM (
SELECT e.node_id,
n.total,
CASE
WHEN date_trunc('day', np.event_time) - INTERVAL '1 day' = date_trunc('day', np.placed_time) THEN 2
WHEN date_trunc('day', np.event_time) - INTERVAL '1 day' > n.placed_time THEN 0
WHEN date_trunc('day', np.event_time) = date_trunc('day', n.placed_time) THEN 1
end as prior_to
FROM events e
JOIN net_parts np on np.node_id = e.node_id
JOIN nets n ON n.id = np.net_id) as summary
GROUP BY node_id, total_stake, prior_to ORDER BY node_id;
Result of the query is:
node_id | node_total_previous_days | node_total_same_day | node_total_previous_day |
---------+--------------------------+---------------------+-------------------------+
6194 | | | 3.00 |
6187 | | 60.00 | 200.00 |
6305 | 150.00 | 569.00 | |
6305 | 150.00 | 569.00 | |
6305 | 150.00 | 569.00 | |
6305 | 150.00 | 569.00 | |
6305 | 150.00 | 569.00 | |
And the question is, how to get grouped result without duplicated values? And to good understand it, why it duplicate that values?
Use group by to determine the rows you want. If you want one row per node_id, then use:
GROUP BY node_id
ORDER BY node_id;
Your additional group by keys are generating more rows. You would see the additional values if you included total_stake and prior_to in the outermost select.