Performance on Degenerated Dimension for drillthrough Action and Processing - sql

There is a lot of information that doesn't fit as Measures and also not have the necessary dimensionality so I decided to integrate this data in the FactTable for a later drillthrough Action (informations like Document Number, Document Line, etc.). So I use the FactTable as a fact dimension (or degenerate dimension as Kimball calls it). The Fact dimension was related after creation with the Measure Group as in the picture below:
The FactTable/Fact dimension has 140.000.000 rows so I decide to use the ROLAP as storage mode trying to avoid the MOLAP processing but now the performance issues are moved in the drillthrough action. All others Dimensions are in MOLAP.
Analysis is istalled on 64x-Server with 98GB RAM and the Memory\TotalMemoryLimit was set to 70% .
I also activate a Profiler when the drillthrough action (over the degenerate dimension in rolap) was performed so I get the SQL-Query. Lots of aggregation and group by - no wondering.
How can I deal with performace in this case so that the drillthrough Action and the processing of degenerate dimension will perform in a timely fashion manner?
UPDATE 13.04
I attached below the execution plan for the query received in Profiler:
SELECT
SUM ( [dbo_FactCdbSAP_Details].[Amount] ) AS Amount,
SUM ( [dbo_FactCdbSAP_Details].[SharedAmount] ) AS SharedAmount,
[dbo_FactCdbSAP_Details].[Pk_id] ,
[dbo_FactCdbSAP_Details].[DocumentNo] ,
[dbo_FactCdbSAP_Details].[DocumentLine] ,
[dbo_FactCdbSAP_Details].[DocumentHeader] ,
[dbo_FactCdbSAP_Details].[DocumentType] ,
[dbo_FactCdbSAP_Details].[Reference] ,
[dbo_FactCdbSAP_Details].[DocumentDate] ,
[dbo_FactCdbSAP_Details].[EntryDate] ,
[dbo_FactCdbSAP_Details].[FiscalPeriod] ,
[dbo_FactCdbSAP_Details].[StornoDocNo] ,
[dbo_FactCdbSAP_Details].[DocumentCurrency] ,
[dbo_FactCdbSAP_Details].[CustomerNumber] ,
[dbo_FactCdbSAP_Details].[EnteredBy] ,
[dbo_FactCdbSAP_Details].[PartnerSegment] ,
[dbo_FactCdbSAP_Details].[PartnerBusinessArea] ,
[dbo_FactCdbSAP_Details].[ItemText] ,
[dbo_FactCdbSAP_Details].[ID_Date] ,
[dbo_FactCdbSAP_Details].[ID_CostCategory] ,
[dbo_FactCdbSAP_Details].[ID_CostCenter] ,
[dbo_FactCdbSAP_Details].[ID_Currency] ,
[dbo_FactCdbSAP_Details].[ID_Branch] ,
[dbo_FactCdbSAP_Details].[ID_Customer] ,
[dbo_FactCdbSAP_Details].[ID_Scenario] ,
[dbo_DimCostCategory_3].[AccountNo] ,
[dbo_DimCostCategory_3].[AccountNameDEU] ,
[dbo_DimCostCategory_3].[AccountNameEng] ,
[dbo_DimCostCategory_3].[AccountType] ,
[dbo_DimCostCategory_3].[AccountSetSAP] ,
[dbo_DimCostCenter_4].[CostCenterNo] ,
[dbo_DimCostCenter_4].[CostCenterName] ,
[dbo_DimCostCenter_4].[CostCenterAliasDEU] ,
[dbo_DimCostCenter_4].[CostCenterAliasENG] ,
[dbo_DimCurrency_5].[CurrencyCode] ,
[dbo_DimCurrency_5].[CurrencyENG] ,
[dbo_DimBranchShare_6].[Branch No] ,
[dbo_DimBranchShare_6].[Branch Name DE] ,
[dbo_DimBranchShare_6].[Branch Name TM1] ,
[dbo_DimBranchShare_6].[Branch Name ENG] ,
[dbo_DimBranchShare_6].[BranchId] ,
[dbo_DimBranchShare_6].[SharePercentage] ,
[dbo_DimBranchShare_6].[Branch Name ASL] ,
[dbo_DimBranchShare_6].[Country] ,
[dbo_DimBranchShare_6].[Currency] ,
[dbo_DimBranchShare_6].[IsSAP] ,
[dbo_DimCustomers_7].[Customer No] ,
[dbo_DimCustomers_7].[Customer Name1] ,
[dbo_DimCustomers_7].[Short Name] ,
[dbo_DimCustomers_7].[Street] ,
[dbo_DimCustomers_7].[Country] ,
[dbo_DimCustomers_7].[Postal Code] ,
[dbo_DimCustomers_7].[Telefon No] ,
[dbo_DimCustomers_7].[Fax TeletexNo] ,
[dbo_DimCustomers_7].[Attending BST] ,
[dbo_DimCustomers_7].[Key Industry Sector] ,
[dbo_DimCustomers_7].[Booking No] ,
[dbo_DimCustomers_7].[Status Inactiv] ,
[dbo_DimCustomers_7].[Company Key] ,
[dbo_DimCustomers_7].[Direct Mailing Forwarder] ,
[dbo_DimCustomers_7].[Direct Mailing BKeeping] ,
[dbo_DimCustomers_7].[Direct Mailing Sales] ,
[dbo_DimCustomers_7].[Direct Mailing Magazines] ,
[dbo_DimCustomers_7].[Customer Name2] ,
[dbo_DimCustomers_7].[Customer Name3] ,
[dbo_DimScenario_8].[ScenarioTypeENG] ,
[dbo_DimDate_2].[Quarter] ,
[dbo_DimDate_2].[Jan-Feb] ,
[dbo_DimDate_2].[Jan-Mrz] ,
[dbo_DimDate_2].[Jan-Apr] ,
[dbo_DimDate_2].[Jan-Mai] ,
[dbo_DimDate_2].[Jan-Jun] ,
[dbo_DimDate_2].[Jan-Jul] ,
[dbo_DimDate_2].[Jan-Aug] ,
[dbo_DimDate_2].[Jan-Sep] ,
[dbo_DimDate_2].[Jan-Okt] ,
[dbo_DimDate_2].[Jan-Nov] ,
[dbo_DimDate_2].[Jan-Dez] ,
[dbo_DimDate_2].[MonthName] ,
[dbo_DimDate_2].[Semester]
FROM (
SELECT
[dbo].[FactCdbSAP_Details].[Pk_id],
[dbo].[FactCdbSAP_Details].[ID_Date],
[dbo].[FactCdbSAP_Details].[ID_Scenario],
[dbo].[FactCdbSAP_Details].[ID_Branch],
[dbo].[FactCdbSAP_Details].[ID_CostCategory],
[dbo].[FactCdbSAP_Details].[ID_CostCenter],
[dbo].[FactCdbSAP_Details].[ID_Customer],
[dbo].[FactCdbSAP_Details].[ID_Currency],
[dbo].[FactCdbSAP_Details].[DocumentNo],
[dbo].[FactCdbSAP_Details].[DocumentLine],
[dbo].[FactCdbSAP_Details].[DocumentHeader],
[dbo].[FactCdbSAP_Details].[DocumentType],
[dbo].[FactCdbSAP_Details].[Reference],
[dbo].[FactCdbSAP_Details].[DocumentDate],
[dbo].[FactCdbSAP_Details].[EntryDate],
[dbo].[FactCdbSAP_Details].[FiscalPeriod],
[dbo].[FactCdbSAP_Details].[StornoDocNo],
[dbo].[FactCdbSAP_Details].[DocumentCurrency],
[dbo].[FactCdbSAP_Details].[CustomerNumber],
[dbo].[FactCdbSAP_Details].[EnteredBy],
[dbo].[FactCdbSAP_Details].[PartnerSegment],
[dbo].[FactCdbSAP_Details].[PartnerBusinessArea],
[dbo].[FactCdbSAP_Details].[ItemText],
[dbo].[FactCdbSAP_Details].[Amount],
[dbo].[FactCdbSAP_Details].[SharedAmount]
FROM [dbo].[FactCdbSAP_Details]
WHERE
id_date >201509
) AS [dbo_FactCdbSAP_Details],
[dbo].[DimCostCategory] AS [dbo_DimCostCategory_3],
[dbo].[DimCostCenter] AS [dbo_DimCostCenter_4],
[dbo].[DimCurrency] AS [dbo_DimCurrency_5],
[dbo].[DimBranchShare] AS [dbo_DimBranchShare_6],
[dbo].[DimCustomers] AS [dbo_DimCustomers_7],
[dbo].[DimScenario] AS [dbo_DimScenario_8],
[dbo].[DimDate] AS [dbo_DimDate_2]
WHERE
[dbo_FactCdbSAP_Details].[ID_Date] = [dbo_DimDate_2].[ID_Date]
AND
[dbo_FactCdbSAP_Details].[ID_CostCategory] = [dbo_DimCostCategory_3].[PK_Cost]
AND
[dbo_FactCdbSAP_Details].[ID_CostCenter] = [dbo_DimCostCenter_4].[Pk_CostCenter]
AND
[dbo_FactCdbSAP_Details].[ID_Currency] = [dbo_DimCurrency_5].[Pk_Currency]
AND
[dbo_FactCdbSAP_Details].[ID_Branch] = [dbo_DimBranchShare_6].[PK_ShareBranch]
AND
[dbo_FactCdbSAP_Details].[ID_Customer] = [dbo_DimCustomers_7].[Pk_Customer]
AND
[dbo_FactCdbSAP_Details].[ID_Scenario] = [dbo_DimScenario_8].[Pk_Scenario]
AND
[dbo_DimCurrency_5].[CurrencyDEU] = 'Lokale Währung'
AND
[dbo_DimScenario_8].[ScenarioTypeDEU] = 'Ist'
AND
[dbo_DimDate_2].[Year] = 2016
AND
[dbo_DimDate_2].[Month] = 2
group by
....

In order to receive a good performance both for drillthrough action and processing the following solution was found and implemented:
I changed the storage mode of the degenerate dimension in MOLAP
AttributeHierarchyOptimizedState=FullyOptimized for all atributes of the degenerate dimension
AttributeHierarchyOrdered=false for the primary key of the degenerate dimension
I implemented the Process Add. A Delta Table Today-Yesterday was created to find out data that are suitable for ProcessAdd (in this scenario old data won't change)
An SSIS Package was created with a DataFlowTask. Inside the DataFlowTask, delta table was set as OLEDB Source and Dimension Processing Task as Destination (this means Incremental Adds direct in the MOLAP Dimension ). The picture below shows that:
In the end will be processed only the affected Cube Partition also with ProcessAdd option.
Many thanks to Greg Galloway for describing ProcessAdd ond large dimension on this post http://www.artisconsulting.com/blogs/greggalloway/2007/4/20/processadd-on-large-dimensions

Related

SQL Partition By Query

Looking for help with a TSQL query.
Lets say you are building Kitchen Cabinets. There are numerous parts to build 1 cabinet and a kitchen in turn will have numerous cabinets. Lets also say that we have 3 different types of cabinets (floor, upper and tall).
We have rack(s) where we want to put all the parts required to build the kitchen. However, we want to keep everything organized. So, Cabinet 1, we want to keep all its parts together.
Our Rack has numerous slots (so many per type of cabinet..upper, floor, tall). The slots can only hold so many pieces before it needs to spill over to the next slot.
So our database has a list of all the parts required per cabinet that is needed. The goal is to associate each part with a rack and slot #. We also know that there are cases where 1 cabinet will need numerous slots and a kitchen may require numerous racks.
How can you write a query to do this. Sample data below:
So The goal (referencing the row number - slot number).
1-1, 2-2, 3-3, 4-3, 5-3, 6-3, 7-3, 8-3, 9-4, 10-4......
Cabinet ID 32 needs 3 slots. So we want to show that you are going to start in slot 3, fill it, move to slot 4, fill it and then move on to slot 5.
I have separate tables that tell me how many slots per slot type are in a rack.
I am using the following so far:
SELECT TOP (100) PERCENT dbo.[CAD-CV-Cabinets].[Cabinet ID] ,
dbo.[CAD-CV-Parts].Width ,
dbo.[CAD-CV-Parts].Length ,
dbo.[CAD-CV-MateriaIs].Name ,
dbo.[CAD-CV-Parts].AutoID ,
dbo.[CAD-CV-Cabinets].[Cabinet Type] ,
CASE WHEN [CAD-CV-Parts].Width > 40
OR [CAD-CV-Parts].Width > 40
OR [CAD-CV-Cabinets].[Cabinet Type] IN(3) THEN 'Tall'
WHEN [CAD-CV-Cabinets].[Cabinet Type] IN(1 , 12 , 4 , 5 , 6 , 7) THEN 'Lower'
WHEN [CAD-CV-Cabinets].[Cabinet Type] IN(2 , 9 , 8 , 10 , 11 , 13 , 14) THEN 'Upper'
ELSE 'No Cart' END AS SlotType,
CASE WHEN [CAD-CV-Parts].Width > 40
OR [CAD-CV-Parts].Width > 40
OR [CAD-CV-Cabinets].[Cabinet Type] IN(3)
THEN ceiling(SUM(2) OVER (PARTITION BY [CAD-CV-Cabinets].[Cabinet ID] ORDER BY dbo.[CAD-CV-Parts].AutoID) / (select min(SlotWidth) from CartDetails where SlotType = 'Tall'))
end TallSlotCount FROM dbo.[CAD-CV-Cabinets]
INNER JOIN
dbo.[CAD-CV-Parts]
ON dbo.[CAD-CV-Cabinets].[Cabinet ID] = dbo.[CAD-CV-Parts].CabinetID
INNER JOIN
dbo.[CAD-CV-MateriaIs]
ON dbo.[CAD-CV-Parts].MaterialID = dbo.[CAD-CV-MateriaIs].ID WHERE ([CAD-CV-Parts].Width > 40
OR [CAD-CV-Parts].Width > 40
OR [CAD-CV-Cabinets].[Cabinet Type] IN(3))
and dbo.[CAD-CV-Parts].DoorID = 0
AND dbo.[CAD-CV-Parts].DrawerID = 0
AND dbo.[CAD-CV-Parts].RollOutID = 0
AND dbo.[CAD-CV-Parts].TopID = 0 ORDER BY dbo.[CAD-CV-Cabinets].[Cabinet ID] , dbo.[CAD-CV-Parts].AutoID;

Return records when subdatasheets values are null

I'm pretty new to SQL and am kind of jumping in at the deep end here.
I'm building a tool from scratch in Excel that utilises and Access Database - and whilst the basic Queries aren't causing me any serious issues - the more complex ones are.
I have four tables. Users, Issues, Votes and Comments.
One user can create many issues, one issue can have many votes and one issue can also have many comments.
I want to create a query that shows a list of issues, with the count of vote_id and comment_id for each
I.e. Issue 1 has 3 votes and 4 comments, and so on - however when the item has zero comments or votes, my query is returning nothing at all
SELECT
users.user_name
, Count(vote.query_id) AS CountOfquery_id
, Count(comments.query_id) AS CountOfquery_id1
, issues.query_id
, issues.query_raised_by
, issues.query_raised_date
, issues.query_summary
, issues.query_status
, issues.query_status_date
, issues.query_detail
, issues.query_response
, issues.query_tag1
, issues.query_tag2
, issues.query_tag3
, issues.query_tag4
FROM
(
(users INNER JOIN issues
ON users.user_id = issues.query_raised_by)
INNER JOIN vote
ON issues.query_id = vote.query_id)
INNER JOIN comments
ON issues.query_id = comments.query_id
GROUP BY
users.user_name
, issues.query_id
, issues.query_raised_by
, issues.query_raised_date
, issues.query_summary
, issues.query_status
, issues.query_status_date
, issues.query_detail
, issues.query_response
, issues.query_tag1
, issues.query_tag2
, issues.query_tag3
, issues.query_tag4;
Is there an easy way to do this? Am I massively overcomplicating the issue?
Basically I want to populate a table in Excel with a list of issues and the number of votes and comments for each, how can I get count to work?
Does doing:
COUNT(vote.query_id) OVER (PARTITION BY users.user_name)
COUNT(comments.query_id) OVER (PARTITION BY users.user_name)
and changing your group statement to say group by users.user_name only give you the result you're after?
#Dekks suggested using Left Join instead of Inner Join - I just googled what the different joins do and it makes a lot more sense now.
Just wanted to post an update for you, in case it helps anyone else :)

Using Intervalmatch without a synthetic key

I'm attempting to use IntervalMatch function to join two tables together a below
InvoiceData:
load Supplier
, SupplierName
, SupplierValue
, Invoice
, InvoiceDate
, DueDate
, OrigInvValue
, OrigDiscValue
, PaymentReference
, PaymentNumber
, PostValue
, Value
, MthInvBal1
, MthInvBal2
, MthInvBal3
, Currency
, ConvRate
, DatabaseName&'.'&Supplier&'.'&Invoice as SupplierInvoice
, DatabaseName as Company
;
SQL Select ****;
CurrencyRates:
Load date(floor([StartDateTime])) as [StartDate]
,date(floor([EndDateTime])) as [EndDate]
,[Currency] as BaseCurrency
,[CADDivision]
,[CHFDivision]
,[EURDivision]
,[GBPDivision]
,[JPYDivision]
,[USDDivision]
,[CADMultiply]
,[CHFMultiply]
,[EURMultiply]
,[GBPMultiply]
,[JPYMultiply]
,[USDMultiply];
SQL SELECT [CR].[StartDateTime]
, [CR].[EndDateTime]
, [CR].[Currency]
, [CR].[CADDivision]
, [CR].[CHFDivision]
, [CR].[EURDivision]
, [CR].[GBPDivision]
, [CR].[JPYDivision]
, [CR].[USDDivision]
, [CR].[CADMultiply]
, [CR].[CHFMultiply]
, [CR].[EURMultiply]
, [CR].[GBPMultiply]
, [CR].[JPYMultiply]
, [CR].[USDMultiply]
FROM [Lookups].[CurrencyRates] [CR];
IntervalMatch:
IntervalMatch (InvoiceDate)
Load distinct [StartDate],[EndDate] Resident CurrencyRates;
From reading the literature, I don't think the should be a synthetic key between the table interval match and the currency rates, however, my data model is still showing this. Is this correct?
You get a synthetic key everytime any two tables are linked with more than one field (in your case StartDateTime and EndDateTime).
Looking at the article from Henric Cronström on Qlik Design Blog (https://community.qlik.com/blogs/qlikviewdesignblog/2013/04/04/intervalmatch) you can read that :
Further, the data model contains a composite key (the FromDate and ToDate fields) which will manifest itself as a QlikView synthetic key. But have no fear. This synthetic key should be there; not only is it correct, but it is also optimal given the data model. You do not need to remove it.
So it seems only natural you'll get that synthetic key.

SSAS Drillthrough in Excel

I have an issue when trying to drill through in excel. I have a fact table of employee terminations set up like the following. Each row is a termination.
For the drillthrough function in SSAS I am using ASSP.GetCustomDrilthroughMDX. Basically created a new action in the cube with the following action expression.
'call ASSP.ExecuteDrillthroughAndFixColumns("'
+ ASSP.GetCustomDrillthroughMDX("
[$Employee].[Employee ID]
, [$Employee].[Company Name]
, [$Employee].[Last Name]
, [$Employee].[First Name]
, [$Employee].[Start Date]
, [$Employee].[Last Day Worked]
, [$Employee].[HR Title]
, [$Employee].[Working Title]
, [$Employee].[HR Status]
, [$Employee].[Location ID]
, [$Department].[Department Number]
, [$Department].[Department Name]
, [$Employee].[Supervisor]
, [$Division].[Division Name]
")
+ '")'
The Condition is
[Measures].CurrentMember IS [Measures].[Involuntary] or
[Measures].CurrentMember IS [Measures].[Voluntary]
When I bring the Desired, Undesired, Voluntary, Involuntary and Other measures into a pivot table and try drill into say the Voluntary, I get the detail of the total amount of employees terminated, not just the Voluntary. For example if 6 employees were Voluntary, and 4 were Involuntary, and I drill into the voluntary I don't get the 6 employees, I get all 10 detail rows. I'm hoping I don't have to create a separate measure group for each type of termination. Any ideas?
So I figured this out.
In the Cube Design, I clicked on each measure in the Termination measure group and went to properties. Under properties expand Source. For NullProcessing property, set it to Preserve. This will preserve the null values in the underlying fact table. By default SSAS treats nulls as zero. Hope this helps someone else.

Abysmal performance using DECRYPTBYKEY in SQL Server 2008 R2

I'm looking at a query that has relatively complex relationship with other tables. Here's what my original query looks like:
SELECT
SE_CUS_DELIVERY.RECEIPT_NAME,
SE_CUS_DELIVERY.ORDER_PHONE,
SE_CUS_DELIVERY.ORDER_ZIP,
SE_CUS_DELIVERY.ORDER_ADDR,
ISNULL(SE_CUS_DELIVERY.DELIV_QTY,SRCHGT.CREQTY),
SE_CUS_DELIVERY.ORDER_HAND
FROM LC_OUT_REQUEST_DETAIL ,
SE_INVENTORY ,
SE_CUSTOMER ,
SRCHGT ,
SE_CUS_DELIVERY
WHERE
LC_OUT_REQUEST_DETAIL.TOTDATE = '20140203'
AND LC_OUT_REQUEST_DETAIL.IO_GB = '021'
AND LC_OUT_REQUEST_DETAIL.LOCCD >= 'A0000'
... A lot of additional joins here
group by SRCHGT.CRDATE + SRCHGT.CRESEQ + SRCHGT.CRESEQ_SEQ + SE_CUS_DELIVERY.DELIV_SEQ ,
SE_CUS_DELIVERY.RECEIPT_NAME ,
SE_CUS_DELIVERY.ORDER_PHONE ,
SE_CUS_DELIVERY.ORDER_ZIP ,
SE_CUS_DELIVERY.ORDER_ADDR ,
ISNULL(SE_CUS_DELIVERY.DELIV_QTY,SRCHGT.CREQTY) ,
... Also a lot of group by's following here
order by LC_OUT_REQUEST_DETAIL.TOTDATE,
LC_OUT_REQUEST_DETAIL.TOT_NO asc,
LC_OUT_REQUEST_DETAIL.TOT_NO_SEQ
To my surprise, it takes about a second to retrieve more than 10,000 rows.
Nevertheless, I've encrypted data in some columns that contains sensitive data, and I modify my select query like so to get the original value:
open Symmetric Key Sym_Key_TestEnc
decryption by certificate Cert_Test
with password = 'somepasswordhere'
GO
SELECT
DECRYPTBYKEY(SE_CUS_DELIVERY.RECEIPT_NAME),
DECRYPTBYKEY(SE_CUS_DELIVERY.ORDER_PHONE),
DECRYPTBYKEY(SE_CUS_DELIVERY.ORDER_ZIP),
DECRYPTBYKEY(SE_CUS_DELIVERY.ORDER_ADDR),
ISNULL(SE_CUS_DELIVERY.DELIV_QTY,SRCHGT.CREQTY),
DECRYPTBYKEY(SE_CUS_DELIVERY.ORDER_HAND)
FROM LC_OUT_REQUEST_DETAIL,
SE_INVENTORY ,
SE_CUSTOMER ,
SRCHGT ,
SE_CUS_DELIVERY
WHERE
LC_OUT_REQUEST_DETAIL.TOTDATE = '20140203'
AND LC_OUT_REQUEST_DETAIL.IO_GB = '021'
AND LC_OUT_REQUEST_DETAIL.LOCCD >= 'A0000'
AND LC_OUT_REQUEST_DETAIL.LOCCD <= 'A9999'
AND LC_OUT_REQUEST_DETAIL.MAT_CD = SE_INVENTORY.MAT_CD
AND LC_OUT_REQUEST_DETAIL.JCOLOR = SE_INVENTORY.JCOLOR
....
group by SRCHGT.CRDATE + SRCHGT.CRESEQ + SRCHGT.CRESEQ_SEQ + SE_CUS_DELIVERY.DELIV_SEQ ,
SE_CUS_DELIVERY.RECEIPT_NAME ,
SE_CUS_DELIVERY.ORDER_PHONE ,
SE_CUS_DELIVERY.ORDER_ZIP ,
SE_CUS_DELIVERY.ORDER_ADDR ,
.......
GO
Close Symmetric key Sym_Key_TestEnc
Now the performance is abysmal. I've been running the same query for more than 5 minutes and it still does not complete.
According to the MSDN, there shouldn't be much issues performance wise
Symmetric encryption and decryption is relatively fast, and is
suitable for working with large amounts of data.
Which leads me to think that I must be doing something wrong. Or MSDN is lying to me, but that's probably not the case.
Is there a way to optimize the data decryption in this process?