Calculate moving average with null values - sql

I have a school graduation data set by year and subgroup and have been provided the numerator and denominator and the single year graduation rate but I also need to calculate a 3 year moving average. I was advised by a statistician that no longer works with us that to do this I needed to get the running total for the numerator for 3 years and the running total for 3 years for the denominator. I understand the math behind it and have checked my work by hand and via excel with a few subgroups. I have also calculated this using T-SQL with no problem so long as there are no null records but I’m struggling with the calculation when there are nulls or 0.
I have tried running the query accounting for null by using NULLIF
ID,
Bldg,
GradClass,
Sbgrp ,
TGrads,
TStus,
Rate,
/*Numerator Running total*/
SUM (TGrads) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) AS NumSum,
/*Denominator Running Total*/
SUM ( TStus) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) AS DenSum,
/*Moving Year Average*/
(
( SUM ( TGrads) OVER ( partition BY DistrictID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) ) / NULLIF ( ( SUM ( TStus) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) ), 0 ) * 100
) AS 3yrAvg
FROM
KResults.DGSRGradBldg
First question, I was provided a record for all subgroups even if they didn’t have students in the subgroup. I want to keep the record so that all subgroups are accounted for within the district and since I know that they didn’t have data, can I substitute the Null values in Tgrads, TStus with a 0? If I do substitute those values with a 0 how can I show the rate as null?
Second question how can I compute the rate with either a null or 0 denominator? I understand you can’t divide by 0 but I want to maintain the record so it’s easy and clear to see that they had no data. How can I do this? When I try to calculate this without accounting for Null I get errors, 1.)Divide by zero error encountered. (8134) and 2.) Null value is eliminated by an aggregate or other SET operation. (8153).
Knowing I can’t divide by 0 or Null I modified my query to include NULLIF and when I do that the query runs with no errors but I don’t get accurate percentage for rates that are below 100%. All my rates are now either 100% or 0 - note the last row, the moving average of 2/3 is not 0.
Here’s what the data looks like if I try to account for nulls my Moving three year average shows as 0. Note the Moving three year Avg Column shows all 0.
ID Bldg Class Sbggrp TGrads TStus Rate NumSum DenSum 3yrAvg
A 1 2014 A1 46 49 93.9 46 49 0
A 1 2015 A1 41 46 89.1 87 95 0
A 1 2016 A1 47 49 95.9 134 144 0
A 1 2017 A1 38 40 95.0 126 135 0
A 1 2018 A1 59 59 98.3 143 148 0
A 1 2014 A2 1 1 100 1 1 100
A 1 2015 A2 1 1 100
A 1 2016 A2 1 1 100
A 1 2017 A2 2 3 66.7 2 3 0
A 1 2018 A2 2 2 100 4 5 0
Any advice would be appreciated but please provide suggestions kindly to this newbie.
Thanks for your time and help.

Answer to question 1: put in the select condition
ISNULL(TGrads,0) AS TGRADS,
ISNULL(TStus,0) AS TSTUS,
Answer to question 2: I'd do this
(CASE WHEN SUM(TStus) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) IS NOT NULL
AND SUM(TStus) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) <>0
THEN (SUM(TGrads) OVER ( partition BY DistrictID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) / (SUM(TStus) OVER ( partition BY ID, Sbgrp ORDER BY GradClass ROWS BETWEEN 2 preceding AND CURRENT row ) ) ) * 100
ELSE NULL END
) AS 3yrAvg
I put null after "ELSE"...You can choose your default value.

Related

Dividing a fixed number over multiple rows by weight

So I have a table that has this structure:
Id - Team - User - Weight
For example: In a team we have the goal to reach 500 calls, so I'd have to spread the 500 calls by weight over each user in the team so I get this structure returned:
Id - Team - User - Weight - # Calls to be made
I know I can do this with a OVER(partition by) and that works perfectly, except for one little detail, there's no way to make half a call.
I'd need this distribution to be without commas.
The basic query:
select Id,Team,User,Weight from CallList
4 ED PEDRO 1
5 RE PEDRO 1
6 PO ROOEIC 0,5
7 PO ROOEIC01 0,5
1 AP APSYSL 0,333333333333333
2 AP APSYSL01 0,333333333333333
3 AP APSYSL02 0,333333333333333
And this is what I would want returned
4 ED PEDRO 1 500
5 RE PEDRO 1 500
6 PO ROOEIC 0,5 250
7 PO ROOEIC01 0,5 250
1 AP APSYSL 0,333333333333333 167
2 AP APSYSL01 0,333333333333333 167
3 AP APSYSL02 0,333333333333333 166
This is an arithmetic problem. The idea is to get a "base" value for each id in a team. Then calculate the excess and incrementally add the excess.
select t.*,
(start_value +
(case when row_number() over (partition by team order by id) <= excess
then 1 else 0
end)
) as calls
from (select t.*,
floor(weight * 500) as start_value,
500 - sum(floor(weight * 500)) over (partition by team) as excess
from t
) t;
This adds the excess from the first row. You seem to want it from the last row. You can achieve that using order by id desc in the row_number().
Simplifying Gordon's logic to remove the Derived Table:
SELECT *
,floor(500 * weight) -- always <= 500
+ CASE -- check if there's a remainder n, if yes split it across n rows
WHEN COUNT(*) -- get a (kind of) random row, using row_number/order you might define a specific row
OVER (PARTITION BY Team
ROWS UNBOUNDED PRECEDING)
<= (500 - SUM(floor(500*weight)) -- remainder calculation
OVER (PARTITION BY Team))
THEN 1
ELSE 0
END
FROM CallList

Conditional SUM in SQL Server 2014

I am using SQL Server 2014. When I was testing my code I noticed a problem.
Assume that max personal hour is 80 hours.
SELECT
lsm.EmployeeName,
pd.absenceDate,
pd.amountInDays * 8 AS [HoursReported],
pd.status,
(SUM(CASE WHEN pd.[status]='App' THEN (pd.amountInDays * 8)
ELSE 0 END) OVER (partition by lsm.[EmployeeName] order by pd.absenceDate)) AS [TotalUsedHours]
( #maxPSHours ) - (sum(
CASE WHEN pd.[status]='App' THEN (pd.amountInDays * 8)
ELSE 0 END)
over (
partition by lsm.[EmployeeName] order by pd.absenceDate)) AS [TotalRemainingHours]
FROM
[LocationStaffMembers] lsm
INNER JOIN
[PersonalDays] pd ON lsm.staffMemberId = pd.staffMemberId
This query returns these results:
EmployeeName AbsenceDate HoursReported Status TotalUsdHrs TotalRemingHrs
X 11/11/2015 4 approved 4 76
X 11/15/2015 8 approved 12 68
X 11/20/2015 2 decline 14 66
X 11/20/2015 2 approved 14 66
So, query works fine for different status. First 2 rows are fine. But when an employee does more than one action in a day (decline, approved etc.), my query only shows the total used and total remaining for the day.
Here is the expected result.
EmployeeName AbsenceDate HoursReported Status TotalUsdHrs TotalRemingHrs
X 11/11/2015 4 approved 4 76
X 11/15/2015 8 approved 12 68
X 11/20/2015 2 decline 12 68
X 11/20/2015 2 approved 14 66
You are doing a cumulative sum that returns results based on the order of AbsenceDate (sum(...) over (partition by ... order by pd.absenceDate). But your last 2 records have the exact same date (11/20/2015) -- at least, according to what you are showing us. This creates an ambiguity.
So, it is absolutely conceivable, and legal, that SQL Server is processing the 2 approved hours row before the 2 declined hours row when calculating the cumulative sum --which would explain your current results--, despite the fact that rows themselves are returned to you in a different order (BTW, consider adding an order by clause to the query, otherwise, the order of the rows themselves are not guaranteed).
If the 2 rows do in fact share the exact same date, you'll have to find a 2nd column to remove the ambiguity and add that to the order by clause in the cumulative sum window function. Maybe you could add a timestamp field that you can order by.
Or maybe you always want the declined status to be considered ahead of the approved status when the AbsenceDate is the same. Here is an example of a query that would do exactly that (notice the changes in the order by clauses):
SELECT
lsm.EmployeeName,
pd.absenceDate,
pd.amountInDays * 8 AS [HoursReported],
pd.status,
(SUM(CASE WHEN pd.[status]='App' THEN (pd.amountInDays * 8)
ELSE 0 END) OVER (partition by lsm.[EmployeeName] order by pd.absenceDate,
case when pd.[status] = 'App' then 1 else 0 end)) AS [TotalUsedHours]
( #maxPSHours ) - (sum(
CASE WHEN pd.[status]='App' THEN (pd.amountInDays * 8)
ELSE 0 END)
over (
partition by lsm.[EmployeeName] order by pd.absenceDate,
case when pd.[status] = 'App' then 1 else 0 end)) AS [TotalRemainingHours]
FROM
[LocationStaffMembers] lsm
INNER JOIN
[PersonalDays] pd ON lsm.staffMemberId = pd.staffMemberId
ORDER BY lsm.[EmployeeName],
pd.absenceDate,
case when pd.[status] = 'App' then 1 else 0 end

How to find the SQL medians for a grouping

I am working with SQL Server 2008
If I have a Table as such:
Code Value
-----------------------
4 240
4 299
4 210
2 NULL
2 3
6 30
6 80
6 10
4 240
2 30
How can I find the median AND group by the Code column please?
To get a resultset like this:
Code Median
-----------------------
4 240
2 16.5
6 30
I really like this solution for median, but unfortunately it doesn't include Group By:
https://stackoverflow.com/a/2026609/106227
The solution using rank works nicely when you have an odd number of members in each group, i.e. the median exists within the sample, where you have an even number of members the rank method will fall down, e.g.
1
2
3
4
The median here is 2.5 (i.e. half the group is smaller, and half the group is larger) but the rank method will return 3. To get around this you essentially need to take the top value from the bottom half of the group, and the bottom value of the top half of the group, and take an average of the two values.
WITH CTE AS
( SELECT Code,
Value,
[half1] = NTILE(2) OVER(PARTITION BY Code ORDER BY Value),
[half2] = NTILE(2) OVER(PARTITION BY Code ORDER BY Value DESC)
FROM T
WHERE Value IS NOT NULL
)
SELECT Code,
(MAX(CASE WHEN Half1 = 1 THEN Value END) +
MIN(CASE WHEN Half2 = 1 THEN Value END)) / 2.0
FROM CTE
GROUP BY Code;
Example on SQL Fiddle
In SQL Server 2012 you can use PERCENTILE_CONT
SELECT DISTINCT
Code,
Median = PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY Value) OVER(PARTITION BY Code)
FROM T;
Example on SQL Fiddle
SQL Server does not have a function to calculate medians, but you could use the ROW_NUMBER function like this:
WITH RankedTable AS (
SELECT Code, Value,
ROW_NUMBER() OVER (PARTITION BY Code ORDER BY VALUE) AS Rnk,
COUNT(*) OVER (PARTITION BY Code) AS Cnt
FROM MyTable
)
SELECT Code, Value
FROM RankedTable
WHERE Rnk = Cnt / 2 + 1
To elaborate a bit on this solution, consider the output of the RankedTable CTE:
Code Value Rnk Cnt
---------------------------
4 240 2 3 -- Median
4 299 3 3
4 210 1 3
2 NULL 1 2
2 3 2 2 -- Median
6 30 2 3 -- Median
6 80 3 3
6 10 1 3
Now from this result set, if you only return those rows where Rnk equals Cnt / 2 + 1 (integer division), you get only the rows with the median value for each group.

SQL query to return rows in multiple groups

I have a SQL table with data in the following format:
REF FIRSTMONTH NoMONTHS VALUE
--------------------------------
1 2 1 100
2 4 2 240
3 5 4 200
This shows a quoted value which should be delivered starting on the FIRSTMONTH and split over NoMONTHS
I want to calculate the SUM for each month of the potential deliveries from the quoted values.
As such I need to return the following result from a SQL server query:
MONTH TOTAL
------------
2 100 <- should be all of REF=1
4 120 <- should be half of REF=2
5 170 <- should be half of REF=2 and quarter of REF=3
6 50 <- should be quarter of REF=3
7 50 <- should be quarter of REF=3
8 50 <- should be quarter of REF=3
How can I do this?
You are trying extract data from what should be a many to many relationship.
You need 3 tables. You should be able to write a JOIN or GROUP BY select statement from there. The tables below don't use the same data values as yours, and are merely intended for a structural example.
**Month**
REF Month Value
---------------------
1 2 100
2 3 120
etc.
**MonthGroup**
REF
---
1
2
**MonthsToMonthGroups**
MonthREF MonthGroupREF
------------------
1 1
2 2
2 3
The first part of this query gets a set of numbers between the start and the end of the valid values
The second part takes each month value, and divides it into the monthly amount
Then it is simply a case of grouping each month, and adding up all of the monthly amounts.
select
number as month, sum(amount)
from
(
select number
from master..spt_values
where type='p'
and number between (select min(firstmonth) from yourtable)
and (select max(firstmonth+nomonths-1) from yourtable)
) numbers
inner join
(select
firstmonth,
firstmonth+nomonths-1 as lastmonth,
value / nomonths as amount
from yourtable) monthly
on numbers.number between firstmonth and lastmonth
group by number

How to track how many times a column changed its value?

I have a table called crewWork as follows :
CREATE TABLE crewWork(
FloorNumber int, AptNumber int, WorkType int, simTime int )
After the table was populated, I need to know how many times a change in apt occurred and how many times a change in floor occurred. Usually I expect to find 10 rows on each apt and 40-50 on each floor.
I could just write a scalar function for that, but I was wondering if there's any way to do that in t-SQL without having to write scalar functions.
Thanks
The data will look like this:
FloorNumber AptNumber WorkType simTime
1 1 12 10
1 1 12 25
1 1 13 35
1 1 13 47
1 2 12 52
1 2 12 59
1 2 13 68
1 1 14 75
1 4 12 79
1 4 12 89
1 4 13 92
1 4 14 105
1 3 12 115
1 3 13 129
1 3 14 138
2 1 12 142
2 1 12 150
2 1 14 168
2 1 14 171
2 3 12 180
2 3 13 190
2 3 13 200
2 3 14 205
3 3 14 216
3 4 12 228
3 4 12 231
3 4 14 249
3 4 13 260
3 1 12 280
3 1 13 295
2 1 14 315
2 2 12 328
2 2 14 346
I need the information for a report, I don't need to store it anywhere.
If you use the accepted answer as written now (1/6/2023), you get correct results with the OP dataset, but I think you can get wrong results with other data.
CONFIRMED: ACCEPTED ANSWER HAS A MISTAKE (as of 1/6/2023)
I explain the potential for wrong results in my comments on the accepted answer.
In this db<>fiddle, I demonstrate the wrong results. I use a slightly modified form of accepted answer (my syntax works in SQL Server and PostgreSQL). I use a slightly modified form of the OP's data (I change two rows). I demonstrate how the accepted answer can be changed slightly, to produce correct results.
The accepted answer is clever but needs a small change to produce correct results (as demonstrated in the above db<>fiddle and described here:
Instead of doing this as seen in the accepted answer COUNT(DISTINCT AptGroup)...
You should do thisCOUNT(DISTINCT CONCAT(AptGroup, '_', AptNumber))...
DDL:
SELECT * INTO crewWork FROM (VALUES
-- data from question, with a couple changes to demonstrate problems with the accepted answer
-- https://stackoverflow.com/q/8666295/1175496
--FloorNumber AptNumber WorkType simTime
(1, 1, 12, 10 ),
-- (1, 1, 12, 25 ), -- original
(2, 1, 12, 25 ), -- new, changing FloorNumber 1->2->1
(1, 1, 13, 35 ),
(1, 1, 13, 47 ),
(1, 2, 12, 52 ),
(1, 2, 12, 59 ),
(1, 2, 13, 68 ),
(1, 1, 14, 75 ),
(1, 4, 12, 79 ),
-- (1, 4, 12, 89 ), -- original
(1, 1, 12, 89 ), -- new , changing AptNumber 4->1->4 ges)
(1, 4, 13, 92 ),
(1, 4, 14, 105 ),
(1, 3, 12, 115 ),
...
DML:
;
WITH groupedWithConcats as (SELECT
*,
CONCAT(AptGroup,'_', AptNumber) as AptCombo,
CONCAT(FloorGroup,'_',FloorNumber) as FloorCombo
-- SQL SERVER doesnt have TEMPORARY keyword; Postgres doesn't understand # for temp tables
-- INTO TEMPORARY groupedWithConcats
FROM
(
SELECT
-- the columns shown in Andriy's answer:
-- https://stackoverflow.com/a/8667477/1175496
ROW_NUMBER() OVER ( ORDER BY simTime) as RN,
-- AptNumber
AptNumber,
ROW_NUMBER() OVER (PARTITION BY AptNumber ORDER BY simTime) as RN_Apt,
ROW_NUMBER() OVER ( ORDER BY simTime)
- ROW_NUMBER() OVER (PARTITION BY AptNumber ORDER BY simTime) as AptGroup,
-- FloorNumber
FloorNumber,
ROW_NUMBER() OVER (PARTITION BY FloorNumber ORDER BY simTime) as RN_Floor,
ROW_NUMBER() OVER ( ORDER BY simTime)
- ROW_NUMBER() OVER (PARTITION BY FloorNumber ORDER BY simTime) as FloorGroup
FROM crewWork
) grouped
)
-- if you want to see how the groupings work:
-- SELECT * FROM groupedWithConcats
-- otherwise just run this query to see the counts of "changes":
SELECT
COUNT(DISTINCT AptCombo)-1 as CountAptChangesWithConcat_Correct,
COUNT(DISTINCT AptGroup)-1 as CountAptChangesWithoutConcat_Wrong,
COUNT(DISTINCT FloorCombo)-1 as CountFloorChangesWithConcat_Correct,
COUNT(DISTINCT FloorGroup)-1 as CountFloorChangesWithoutConcat_Wrong
FROM groupedWithConcats;
ALTERNATIVE ANSWER
The accepted-answer may eventually get updated to remove the mistake. If that happens I can remove my warning but I still want leave you with this alternative way to produce the answer.
My approach goes like this: "check the previous row, if the value is different in previous row vs current row, then there is a change". SQL doesn't have idea or row order functions per se (at least not like in Excel for example; )
Instead, SQL has window functions. With SQL's window functions, you can use the window function RANK plus a self-JOIN technique as seen here to combine current row values and previous row values so you can compare them. Here is a db<>fiddle showing my approach, which I pasted below.
The intermediate table, showing the columns which has a value 1 if there is a change, 0 otherwise (i.e. FloorChange, AptChange), is shown at the bottom of the post...
DDL:
...same as above...
DML:
;
WITH rowNumbered AS (
SELECT
*,
ROW_NUMBER() OVER ( ORDER BY simTime) as RN
FROM crewWork
)
,joinedOnItself AS (
SELECT
rowNumbered.*,
rowNumberedRowShift.FloorNumber as FloorShift,
rowNumberedRowShift.AptNumber as AptShift,
CASE WHEN rowNumbered.FloorNumber <> rowNumberedRowShift.FloorNumber THEN 1 ELSE 0 END as FloorChange,
CASE WHEN rowNumbered.AptNumber <> rowNumberedRowShift.AptNumber THEN 1 ELSE 0 END as AptChange
FROM rowNumbered
LEFT OUTER JOIN rowNumbered as rowNumberedRowShift
ON rowNumbered.RN = (rowNumberedRowShift.RN+1)
)
-- if you want to see:
-- SELECT * FROM joinedOnItself;
SELECT
SUM(FloorChange) as FloorChanges,
SUM(AptChange) as AptChanges
FROM joinedOnItself;
Below see the first few rows of the intermediate table (joinedOnItself). This shows how my approach works. Note the last two columns, which have a value of 1 when there is a change in FloorNumber compared to FloorShift (noted in FloorChange), or a change in AptNumber compared to AptShift (noted in AptChange).
floornumber
aptnumber
worktype
simtime
rn
floorshift
aptshift
floorchange
aptchange
1
1
12
10
1
0
0
2
1
12
25
2
1
1
1
0
1
1
13
35
3
2
1
1
0
1
1
13
47
4
1
1
0
0
1
2
12
52
5
1
1
0
1
1
2
12
59
6
1
2
0
0
1
2
13
68
7
1
2
0
0
Note instead of using the window function RANK and JOIN, you could use the window function LAG to compare values in the current row to the previous row directly (no need to JOIN). I don't have that solution here, but it is described in the Wikipedia article example:
Window functions allow access to data in the records right before and after the current record.
If I am not missing anything, you could use the following method to find the number of changes:
determine groups of sequential rows with identical values;
count those groups;
subtract 1.
Apply the method individually for AptNumber and for FloorNumber.
The groups could be determined like in this answer, only there's isn't a Seq column in your case. Instead, another ROW_NUMBER() expression could be used. Here's an approximate solution:
;
WITH marked AS (
SELECT
FloorGroup = ROW_NUMBER() OVER ( ORDER BY simTime)
- ROW_NUMBER() OVER (PARTITION BY FloorNumber ORDER BY simTime),
AptGroup = ROW_NUMBER() OVER ( ORDER BY simTime)
- ROW_NUMBER() OVER (PARTITION BY AptNumber ORDER BY simTime)
FROM crewWork
)
SELECT
FloorChanges = COUNT(DISTINCT FloorGroup) - 1,
AptChanges = COUNT(DISTINCT AptGroup) - 1
FROM marked
(I'm assuming here that the simTime column defines the timeline of changes.)
UPDATE
Below is a table that shows how the distinct groups are obtained for AptNumber.
AptNumber RN RN_Apt AptGroup (= RN - RN_Apt)
--------- -- ------ ---------
1 1 1 0
1 2 2 0
1 3 3 0
1 4 4 0
2 5 1 4
2 6 2 4
2 7 3 4
1 8 5 => 3
4 9 1 8
4 10 2 8
4 11 3 8
4 12 4 8
3 13 1 12
3 14 2 12
3 15 3 12
1 16 6 10
… … … …
Here RN is a pseudo-column that stands for ROW_NUMBER() OVER (ORDER BY simTime). You can see that this is just a sequence of rankings starting from 1.
Another pseudo-column, RN_Apt contains values produces by the other ROW_NUMBER, namely ROW_NUMBER() OVER (PARTITION BY AptNumber ORDER BY simTime). It contains rankings within individual groups of identical AptNumber values. You can see that, for a newly encountered value, the sequence starts over, and for a recurring one, it continues where it stopped last time.
You can also see from the table that if we subtract RN from RN_Apt (could be the other way round, doesn't matter in this situation), we get the value that uniquely identifies every distinct group of same AptNumber values. You might as well call that value a group ID.
So, now that we've got these IDs, it only remains for us to count them (count distinct values, of course). That will be the number of groups, and the number of changes is one less (assuming the first group is not counted as a change).
add an extra column changecount
CREATE TABLE crewWork(
FloorNumber int, AptNumber int, WorkType int, simTime int ,changecount int)
increment changecount value for each updation
if want to know count for each field then add columns corresponding to it for changecount
Assuming that each record represents a different change, you can find changes per floor by:
select FloorNumber, count(*)
from crewWork
group by FloorNumber
And changes per apartment (assuming AptNumber uniquely identifies apartment) by:
select AptNumber, count(*)
from crewWork
group by AptNumber
Or (assuming AptNumber and FloorNumber together uniquely identifies apartment) by:
select FloorNumber, AptNumber, count(*)
from crewWork
group by FloorNumber, AptNumber