Performance issue with SQL Query - sql

I am pulling data from two tables-Forecast and Orders to compute sales forecast accuracy.
Steps I am taking:
Identifying all exhaustive combinations of product-region-demand month b/w both data sets...let's call this (1)
Identifying different forecast snapshots in forecast data... let's call this (2)
Performing a cross-join of (1) and (2)...let's call this (3)
Performing the "SUMIF()" equivalent on the lines from (3) for both orders and forecast. For example, if I am comparing forecast to actual orders for February,
Jan "INDPOR" Forecast---> For some Product/Region-Feb Delivery Combination: February Forecast (generated in January) vs. Orders booked after Jan 1st with a delivery schedule in Feb
Feb "INDPOR" Forecast---> For the same Product/Region-Feb Delivery Date Combination: February Forecast (generated in February) vs. Orders booked after Jan 27th* with a delivery schedule in Feb
Note 1: Generating multiple forecasts for the same month
Note 2: Fiscal calendar definitions used; that is why Feb starts on Jan 27th
Output is generating correctly. But, it is painfully slow (1 hour +). Please help me fine-tune this and make it faster as I will need to use this for larger data sets too.
Other Details:
I am running this on SQL Server 2014 locally from my desktop.Uploading this using the SQL data import wizard into SQL from an Excel file currently
Input Forecast data: ForecastAggTable
Input Orders data: OrderAggTable
Input and Output Files
Code:
Select *
from
(
Select *,
(Select isnull(sum([Forecast Qty]),0) from ForecastAggTable t2 where t2.LOB=D.LOB and
t2.[Demand Month]=D.[Demand Month] and t2.Class=D.Class
and t2.[Item Type]=D.[Item Type] and t2.[LoB Region]=D.[LoB Region] and
t2.[Key Account]=D.[Key Account] and t2.Country=D.Country
and t2.[Master Customer]=D.[Master Customer] and t2.[INDPOR Version]=D.[INDPOR Version])[Forecast Qty],
(
Select isnull(sum([Order Qty]),0) from OrderAggTable t1 where t1.LOB=D.LOB and
t1.[SAD Month]=D.[Demand Month] and t1.Class=D.Class
and t1.[Item Type]=D.[Item Type] and t1.[LoB Region]=D.[LoB Region] and
t1.[Key Account]=D.[Key Account] and t1.Country=D.Country
and t1.[Master Customer]=D.[Master Customer] and t1.[Book Date]>=D.[INDPOR Timestamp]
)[SAD-OrderQty],
(
Select isnull(sum([Order Revenue]),0) from OrderAggTable t1 where t1.LOB=D.LOB and
t1.[SAD Month]=D.[Demand Month] and t1.Class=D.Class
and t1.[Item Type]=D.[Item Type] and t1.[LoB Region]=D.[LoB Region] and
t1.[Key Account]=D.[Key Account] and t1.Country=D.Country
and t1.[Master Customer]=D.[Master Customer] and t1.[Book Date]>=D.[INDPOR Timestamp]
)[SAD-OrderRevenue],
(
Select isnull(sum([Order Qty]),0) from OrderAggTable t1 where t1.LOB=D.LOB and
t1.[RDD Month]=D.[Demand Month] and t1.Class=D.Class
and t1.[Item Type]=D.[Item Type] and t1.[LoB Region]=D.[LoB Region] and
t1.[Key Account]=D.[Key Account] and t1.Country=D.Country
and t1.[Master Customer]=D.[Master Customer] and t1.[Book Date]>=D.[INDPOR Timestamp]
)[RDD-OrderQty],
(
Select isnull(sum([Order Revenue]),0) from OrderAggTable t1 where t1.LOB=D.LOB and
t1.[RDD Month]=D.[Demand Month] and t1.Class=D.Class
and t1.[Item Type]=D.[Item Type] and t1.[LoB Region]=D.[LoB Region] and
t1.[Key Account]=D.[Key Account] and t1.Country=D.Country
and t1.[Master Customer]=D.[Master Customer] and t1.[Book Date]>=D.[INDPOR Timestamp]
)[RDD-OrderRevenue]
from
(
Select distinct LOB,[INDPOR Version],[INDPOR Timestamp],[Demand Month],
[Demand Quarter],[Min Date],Class,[Item Type],[Offer PF],
[LoB Region],[Key Account],Country,[Master Customer]
from
(
Select V.LOB,V.[SAD Month][Demand Month],V.[SAD Quarter][Demand Quarter],V.[SAD Min Date][Min Date],V.Class,
[Item Type],[Offer PF],[LoB Region],[Key Account],Country,[Master Customer]
from OrderAggTable V
union
(
Select Z.LOB,Z.[RDD Month][Demand Month],Z.[RDD Quarter][Demand Quarter],Z.[RDD Min Date][Min Date],Z.Class,
[Item Type],[Offer PF],[LoB Region],[Key Account],Country,[Master Customer]
from OrderAggTable Z
)
union
(
Select LOB,[Demand Month],[Demand Quarter],[Min Date],Class[Class],[Item Type],[Offer PF],[LoB Region],
[Key Account],Country,[Master Customer] from ForecastAggTable
)
)A
cross join
(
select distinct [INDPOR Version],[INDPOR Timestamp]
from ForecastAggTable
)B
)D
where [Min Date]>=[INDPOR Timestamp]
)E
where ([SAD-OrderQty] + [RDD-OrderQty] + [Forecast Qty]<>0)

How about simplifying, and doing less passes of your tables.
In this example I do two scans of the forecast table, one for the distinct, and one for the union, and one scan of the orders table.
with cte as
(
select distinct [INDPOR Version],[INDPOR Timestamp]
from ForecastAggTable
)
,cte2 as
(
Select
V.LOB
,iif(DUP=0,V.[SAD Month] ,V.[RDD Month] ) [Demand Month]
,iif(DUP=0,V.[SAD Quarter] ,V.[RDD Quarter] ) [Demand Quarter]
,iif(DUP=0,V.[SAD Min Date] ,V.[RDD Min Date] ) [Min Date]
,V.[Book Date]
,V.Class
,V.[Item Type]
,V.[Offer PF]
,V.[LoB Region]
,V.[Key Account]
,V.Country
,V.[Master Customer]
,null [INDPOR Version]
,null [Forecast Qty]
,iif(DUP=0,v.[Order Qty] ,null ) [SAD-OrderQty]
,iif(DUP=0,V.[Order Revenue] ,null ) [SAD-OrderRevenue]
,iif(DUP=1,V.[Order Qty] ,null ) [RDD-OrderQty]
,iif(DUP=1,V.[Order Revenue] ,null ) [RDD-OrderRevenue]
from OrderAggTable V
cross join (select dup from (values (0),(1))a(dup)) a
union all
Select
LOB
,[Demand Month]
,[Demand Quarter]
,[Min Date]
,[Min Date]
,Class
,[Item Type]
,[Offer PF]
,[LoB Region]
,[Key Account]
,Country
,[Master Customer]
,[INDPOR Version]
,[Forecast Qty]
,null[SAD-OrderQty]
,null[SAD-OrderRevenue]
,null[RDD-OrderQty]
,null[RDD-OrderRevenue]
from ForecastAggTable
)
select
cte2.LOB
,cte.[INDPOR Version]
,cte.[INDPOR Timestamp]
,cte2.[Demand Month]
,cte2.[Demand Quarter]
,cte2.[Min Date]
,cte2.Class
,cte2.[Item Type]
,cte2.[Offer PF]
,cte2.[LoB Region]
,cte2.[Key Account]
,cte2.Country
,cte2.[Master Customer]
,isnull(sum(cte2.[Forecast Qty] ),0) [Forecast Qty]
,isnull(sum(cte2.[SAD-OrderQty] ),0) [SAD-OrderQty]
,isnull(sum(cte2.[SAD-OrderRevenue]) ,0) [SAD-OrderRevenue]
,isnull(sum(cte2.[RDD-OrderQty] ),0) [RDD-OrderQty]
,isnull(sum(cte2.[RDD-OrderRevenue]) ,0) [RDD-OrderRevenue]
from cte2
inner join cte
on cte2.[Book Date]>=cte.[INDPOR Timestamp]
where isnull(cte2.[INDPOR Version],cte.[INDPOR Version])=cte.[INDPOR Version]
group by
cte2.LOB
,cte2.[Demand Month]
,cte2.[Demand Quarter]
,cte2.[Min Date]
,cte2.Class
,cte2.[Item Type]
,cte2.[Offer PF]
,cte2.[LoB Region]
,cte2.[Key Account]
,cte2.Country
,cte2.[Master Customer]
,cte.[INDPOR Version]
,cte.[INDPOR Timestamp]
having
isnull(sum(cte2.[Forecast Qty] ),0) +
isnull(sum(cte2.[SAD-OrderQty] ),0) +
isnull(sum(cte2.[RDD-OrderQty] ),0)
!=0

Related

Why does my record count increase as my denominator for a division performed in a nested query increases?

In my query I'm trying to calculate the total overtime worked by employees by dividing basic salary by 173(hours in a month) which gives me the hourly rate then dividing the total overtime amount of employee by the hourly rate. To my surprise the record count increases as the number of hours in a month increases, should it not decrease?
Here's my script:
select 'Employees that worked more overtime' [CAAT],*
from ( select H.[Month]
,H.[Employee Code]
,H.[Department]
,H.[Job title]
,H.[Surname]
,H.[Full Names]
,H.[Basic Salary]
,R.[Overtime]
,H.[Hourly Rate]
,round(R.[Overtime] / H.[Hourly Rate],2) [Overtime Hours]
from (select [Month]
,[Employee Code]
,Department
,[Job title]
,[Surname]
,[Full Names]
,nullif(convert(money,[Amount]),0.00) [Basic Salary]
,nullif(round(convert(money,[Amount]) / 173,2),0.00) [Hourly Rate]
from [Salary DB]
where [Field Desc] = 'ED01-Basic Salary') H
left join
(select [Month]
,[Employee Code]
,nullif(sum(convert(money,[Amount])),0.00) [Overtime]
from [Salary DB]
where [Field Desc] in ('ED02-O/Time 1.5','ED02-O/Time 2.0','ED42-Sunday Pay')
group by [Month]
,[Employee Code]) R
on H.[Employee Code] = R.[Employee Code]
and H.[Month] = R.[Month]) [Data]
where [Overtime Hours] > '40'
Order by [Employee Code], [Month] Desc

Meeting 2 conditions in 2 different columns

I'm trying to run a query where I need very specific conditions to be met:
Sales code is All
Item has Original Price flag set
Item has a price with no Original Price flag set that is the same as the Price with Original Price flag set
Price without Original price flag set must be created after the price with Original price flag
Currently I am using the following query to get the information I need;
select [item no_], [variant code],[unit price including vat],
[original price], [Starting Date], [Ending Date] from [Sales Price]
where [Sales Code] = 'all'
and [Ending Date] = '1753-01-01 00:00:00.000'
This is the example result:
1 means Original Price flag is set and 0 means it is not
The result I need from this query would be to only show these two:
I am assuming you are working with SQL Server as your current query syntax suggests.
If, so you can use lag() :
select sp.*
from (select sp.*,
lag([original price]) over (partition by [item no_] order by [Starting Date]) as prev_price
from [Sales Price] sp
where [Sales Code] = 'all'
) sp
where ([original price] = 1 or prev_price = 1);
let me know if you need me to explain; else its pretty straight forward.
select a.*
from (
select [item no_]
, [variant code]
,[unit price including vat]
, [original price]
, [Starting Date]
, [Ending Date]
,Column_Test = case when ( [original price] = 1 and [original price] = 0 ) and ([Starting Date]<[Ending Date]) then 1 else 0 end
from [Sales Price]
where [Sales Code] = 'all'
and [Ending Date] = '1753-01-01 00:00:00.000'
) a
where Column_Test = 1

Script help - count , total and avg

I have the following script, trying to count how many distinct customers are there , how many distinct orders , what is total of all orders under £15 and its's avg, total of orders above £20 and it's avg.
with consignments as
(select
[Sell-to Customer No_],
[Convert-to Document No_],
ic.[Shipping Agent Service Code],
[Pick Completed DateTime] as [Shipped DateTime],
ROUND((ic.[Amount Including VAT] + ic.Postage + ic.[Gift Wrap Price] +
ic.[Handling Fee] + ic.[Personalisation Fee]),2) as [Document Amount]
from dbo.[Temp$Consignment] ic inner join [dbo].
[Temp$Order] oh
on ic.[Owner Header GuID]=oh.[Order Guid]
where ic.[Shipping Agent Service Code]='secstan' and ic.[Pick Completed
DateTime] >= '2016-11-01T00:00:00.000' AND
ic.[Pick Completed DateTime] <= '2016-11-30T23:59:55.000' ),summary as
(select *,CASE WHEN [Document Amount] > 15 THEN 1 ELSE 0 END as 'Over15'
from consignments )select * from summary
I have working script like below, but as I am new to sql I am bit confused how to convert above script to below.
select amountclass,[Shipping Agent Service Code],
count(distinct [Sell-to Customer No_]) as Total_customers,
count(*) as Total_orders,
sum([Amount]) as total_revenue,
avg([Amount] * 1.0) as AOV
from
(
select [Sell-to Customer No_], oh.[Original Order No_], [Amount],ic.
[Shipping Agent Service Code],
case when [Amount] <= 20 then 'Under_20'
else 'Over_20'
end as amountclass
from [TBW_BI].[dbo].[Temp$Order] oh INNER JOIN [TBW_BI].[dbo].
[Temp$Consignment] IC
ON IC.[Owner Header GuID]=OH.[Order Guid]
where[order date] >= '2016-09-01' AND
[order date] <= '2016-09-30' AND [COUNTRY]='UNITED KINGDOM' and
[document type] like 'ord%' and ic.[Shipping Agent Service
Code]='secstan'
) dt
group by amountclass,[Shipping Agent Service Code]
order by amountclass,[Shipping Agent Service Code]
It looks like you have a query with a 2 Common Table Expressions (CTE) and you want to convert the CTE to a derived table. First, we can eliminate one of the CTEs.
summary as
(select *,CASE WHEN [Document Amount] > 15 THEN 1 ELSE 0 END as 'Over15'
from consignments )
select * from summary
The CTE for summary is unnecessary. We can convert that code to this:
select *,CASE WHEN [Document Amount] > 15 THEN 1 ELSE 0 END as 'Over15'
from consignments
Now, instead of using the consignments CTE; we can copy the sql code within it to create a derived table.
select *,CASE WHEN [Document Amount] > 15 THEN 1 ELSE 0 END as 'Over15'
from (
select [Sell-to Customer No_],
[Convert-to Document No_],
ic.[Shipping Agent Service Code],
[Pick Completed DateTime] as [Shipped DateTime],
ROUND((ic.[Amount Including VAT] + ic.Postage + ic.[Gift Wrap Price] +
ic.[Handling Fee] + ic.[Personalisation Fee]),2) as [Document Amount]
from dbo.[Temp$Consignment] ic
inner join [dbo].[Temp$Order] oh on ic.[Owner Header GuID]=oh.[Order Guid]
where ic.[Shipping Agent Service Code]='secstan'
and ic.[Pick Completed
DateTime] >= '2016-11-01T00:00:00.000'
AND
ic.[Pick Completed DateTime] <= '2016-11-30T23:59:55.000' ) as t

Workaround for PIVOT statement

I have this query, is taking like 2 minutes to resolve, I need to find a workaround, I know that UNPIVOT has a better solution using CROSS APPLY, is there anything similar for PIVOT?
SELECT [RowId], [invoice date], [GL], [Entité], [001], [Loc], [Centre Cout], [Compte_1], [Interco_1], [Futur_1], [Department], [Division], [Compagnie], [Localisation], [Centre/Cout], [Compte], [Interco], [Futur], [Account], [Mobile], [Last Name], [First Name], [license fee], [GST], [HST], [PST], [Foreign Tax], [Sales Tax License], [Net Total], [Total], [ServiceType], [Oracle Cost Center], [CTRL], [EXPENSE], [Province]
FROM
(SELECT fd.[RowId], fc.[ColumnName], fd.[Value]
FROM dbo.FileData fd
INNER JOIN dbo.[FileColumn] fc
ON fc.[FileColumnId] = fd.[FileColumnId]
WHERE FileId = 1
AND TenantId = 1) x
PIVOT
(
MAX(Value)
FOR [ColumnName] IN ( [invoice date], [GL], [Entité], [001], [Loc], [Centre Cout], [Compte_1], [Interco_1], [Futur_1], [Department], [Division], [Compagnie], [Localisation], [Centre/Cout], [Compte], [Interco], [Futur], [Account], [Mobile], [Last Name], [First Name], [license fee], [GST], [HST], [PST], [Foreign Tax], [Sales Tax License], [Net Total], [Total], [ServiceType], [Oracle Cost Center], [CTRL], [EXPENSE], [Province])
) AS p
Pivots are great, but so are Conditional Aggregations. Also, there would be no datatype conficts or conversions necessary
SELECT [RowId]
,[invoice date] = max(case when [FileColumnId] = ??? then Value end)
,[GL] = max(case when [FileColumnId] = ??? then Value end)
,... more fields
FROM dbo.FileData fd
WHERE FileId = 1
AND TenantId = 1
Group By [RowId]
EDIT
You could add back the join to make it more readable.

Find the most recent row of a group

I know there is no "last" row so I hope I'm clear that isn't what I'm really looking for. I want to select the rows in a table if the value of one particular field is the last alphabetically. I'll try my best to draw it out below. I'm a bit of a novice so please bear with me...
TABLE
[Order Number], [Delivery Date], [Order Qty], [Shipped Qty], [Bill To], [Ship To], [Invoice Number]
There are many times when we will reissue invoices and that invoice number will increment by a letter. This will also update additional field values as well. Below is a typical set with multiples invoices...
'987654', '2014-05-01 00:00:00', '100', '90', 'BillToXYZ', 'ShipToXYZ', '987654A' - NEW RECORD -
'987654', '2014-05-01 00:00:00', '-100', '-90', 'BillToXYZ', 'ShipToXYZ', '987654B' - NEW RECORD -
'987654', '2014-05-01 00:00:00', '100', '100', 'BillToXYZ', 'ShipToNEWSHIPTOLOCATION', '987654C' - NEW RECORD -
'987654', '2014-05-01 00:00:00', '10', '10', 'BillToXYZ', '2ndNEWSHIPTOLOCATION', '987654D' - NEW RECORD -
What I need is to query all the above fields and only bring back those where the [Invoice Number] is the last(alphabetically) (in this case 987654D) but also have it SUM the values of the [Order Qty] and [Shipped Qty] for all of the records regardless of [Invoice Number].
If I can provide any additional information please let me know. Thank you in advance.
It possible to use the ROW_NUMBER function to get the last row in a group setting the ORDER BY descending and then using a filter to get the row with the value 1.
The SUM and MAX with windowing help to get the other aggregate values.
WITH D AS (
SELECT [Order Number]
, [Delivery Date]
, SUM([Order Qty]) OVER (PARTITION BY [Order Number]) [Total Order Qty]
, [Total Shipped Qty]
= SUM([Shipped Qty]) OVER (PARTITION BY [Order Number])
, [Bill To]
, [Ship To]
, [Last Invoice Number]
= MAX([Invoice Number]) OVER (PARTITION BY [Order Number])
, ID = ROW_NUMBER() OVER (PARTITION BY [Order Number]
ORDER BY [Invoice Number] DESC)
FROM Table1
)
SELECT [Order Number]
, [Delivery Date]
, [Total Order Qty]
, [Total Shipped Qty]
, [Bill To]
, [Ship To]
, [Last Invoice Number]
FROM D
WHERE ID = 1
SQLFiddle demo
Looks to me like you can use a GROUP BY and aggregates as suggested by other answers, but put that in a subquery so that you can link your 'main' table to it and get the detail values for the latest invoice. Does the following code do what you're looking for?
CREATE TABLE #Table
(
OrderNumber int,
DeliveryDate datetime,
OrderQty int,
ShippedQty int ,
BillTo varchar(10),
ShipTo varchar(100),
InvoiceNumber varchar(20)
)
INSERT INTO #Table
(
OrderNumber,
DeliveryDate,
OrderQty,
ShippedQty,
BillTo,
ShipTo,
InvoiceNumber
)
SELECT '987654', '2014-05-01 00:00:00', '100', '90', 'BillToXYZ', 'ShipToXYZ', '987654A' UNION
SELECT '987654', '2014-05-01 00:00:00', '-100', '-90', 'BillToXYZ', 'ShipToXYZ', '987654B' UNION
SELECT '987654', '2014-05-01 00:00:00', '100', '100', 'BillToXYZ', 'ShipToNEWSHIPTOLOCATION', '987654C' UNION
SELECT '987654', '2014-05-01 00:00:00', '10', '10', 'BillToXYZ', '2ndNEWSHIPTOLOCATION', '987654D'
SELECT t.*, s.TotalOrderQty, s.TotalShippedQty
FROM
#Table t
INNER JOIN
(
SELECT
OrderNumber,
MAX(InvoiceNumber) LastInvoice,
SUM(OrderQty) TotalOrderQty,
SUM(ShippedQty) TotalShippedQty
FROM #Table
GROUP BY OrderNumber
) s ON
t.OrderNumber = s.OrderNumber AND
t.InvoiceNumber = s.LastInvoice
DROP TABLE #Table
Is this what you're going after?
EDIT 2: Got clarification from OP. Have edited my solution to bring up aggregation for complete order but only the line item information of the latest invoice. Fiddle here: http://sqlfiddle.com/#!3/08bb0/6
SELECT t1.[Order Number],
t2.[Max Invoice Number],
t1.[Ship To],
t2.[Sum Order Qty],
t2.[Sum Shipped Qty]
FROM Table1 t1 INNER JOIN
(SELECT [Order Number],
MAX([Invoice Number] ) AS [Max Invoice Number],
SUM([Order Qty]) AS [Sum Order Qty],
SUM([Shipped Qty]) AS [Sum Shipped Qty]
FROM Table1
GROUP BY [Order Number]) t2
ON t1.[Invoice Number] = t2.[Max Invoice Number]
ORDER BY t1.[Order Number];
You can add more columns from the D invoice by placing them in the SELECT list from t1, as I did with [Ship To].
EDIT 1: Added grouping so whole table is considered. To be fair, Martin K. posted this answer first.