retrieve large amount of data from multiple tables within seconds - sql

The following query retrieves data from multiple tables.
There is a problem, as the data increases the amount of time to retrieve the data increases too ( the number of rows has reached to 20 thousand and it takes more than a minute to show the result).
i have used stored procedure but the result was same. Any Solution ?
Select TransactionDate,
T.InvoiceNo,
T.TotalAmount,
PCost.TotalCostPrice ,
(T.TotalAmount - PCost.TotalCostPrice) as TProfit,
P.PaidAmount, P.Balance,
CONCAT(Lastname, ', ', Firstname, ' ', Middlename) as Cashier,
ISNULL(P.DiscountAmount,0),
ISNULL(P.ProfitAmount,0),
CustomerInfo
FROM [Transaction] T
INNER JOIN TransactionPayment P ON T.InvoiceNo = P.InvoiceNo
INNER JOIN Employee E ON T.EmployeeID = E.EmployeeID
LEFT OUTER JOIN (
SELECT SUM(p.CostPrice*tl.Quantity) as TotalCostPrice,
tl.InvoiceNo
FROM Product p
INNER JOIN TransactionLineItem tl ON p.ProductID = tl.ProductID
INNER JOIN [Transaction] T ON T.InvoiceNo = tl.InvoiceNo
WHERE TransactionDate >= #DateFrom AND TransactionDate <= #DateTo
GROUP BY tl.InvoiceNo
) as PCost ON PCost.InvoiceNo = T.InvoiceNo
WHERE TransactionDate >= #DateFrom AND TransactionDate <= #DateTo

Try this:
create index IX_Transaction_TransactionDate
on dbo.[Transaction] (TransactionDate, InvoiceNo);

Related

Order by on a nested query

Is their a way to order this by the Time column? I am not sure how to do this. The time is the schedule and I just need it to go from the morning to the evening.
Can I just nest another select statement and use that?
Thank you.
SELECT
DoseLevel,
LastName,
FirstName,
DOB,
EMPLID,
Time,
(
SELECT v.ColorCode
FROM ABCDocumentation cd1
LEFT JOIN ABCDocumentation cd2ON cd1.ABCDocumentationID = cd2.PairID
LEFT JOIN Medicine v ON v.MedicineID = cd1.MedicineID
LEFT JOIN Manufacturers mfg ON v.MFG_Seq = mfg.MFG_Seq
WHERE cd2.ABCDocumentationID = dt.ABCDocumentationID
) AS ParentColorCode,
(
SELECT mfg.Description
FROM ABCDocumentation cd1
LEFT JOIN ABCDocumentation cd2 ON cd1.ABCDocumentationID = cd2.PairID
LEFT JOIN Medicine v ON v.MedicineID = cd1.MedicineID
LEFT JOIN Manufacturers mfg ON v.MFG_Seq = mfg.MFG_Seq
WHERE cd2.ABCDocumentationID = dt.ABCDocumentationID
) AS ParentManuDesc
FROM
(
SELECT
cd.DoseLevel,
e.LastName,
e.FirstName,
e.DOB,
cvse.EMPLID,
cvse.AdminScheduleSlotsEmployeeID,
cd.ABCDocumentationID,
cvss.Time,
cd.ModifyDate AS 'StartTime'
FROM ABCAdminSchedule cvs
LEFT JOIN ABCAdminScheduleSlots cvss ON cvs.AdminScheduleID = cvss.AdminScheduleID
LEFT JOIN ABCAdminScheduleSlotsEmployee cvse ON cvss.AdminScheduleSlotsID = cvse.AdminScheduleSlotsID
LEFT JOIN ABCDocumentation cd ON cvse.AdminScheduleSlotsEmployeeID = cd.AdminScheduleSlotsEmployeeID
LEFT JOIN Employee e ON cvse.EmplID = e.EMPLID
WHERE CAST(TIME AS Date) = CAST(GETDATE() AS Date) AND CampusID = '06'
AND cvse.AdminScheduleSlotsEmployeeID IS NOT NULL
) dt
First off, there is no need for the derived table dt, as you are not doing any further processing.
Secondly, you can combine the two correlated subqueries into one with an APPLY.
Thirdly, conversions on columns can cause performance issues, so you can change the date check to a half-open interval, converting just GETDATE().
Finally you can add at the end an ORDER BY clause to sort.
SELECT
cd.DoseLevel,
e.LastName,
e.FirstName,
e.DOB,
cvse.EMPLID,
cvss.Time,
Parent.ColorCode,
Parent.Description
FROM ABCAdminSchedule cvs
LEFT JOIN ABCAdminScheduleSlots cvss ON cvs.AdminScheduleID = cvss.AdminScheduleID
LEFT JOIN ABCAdminScheduleSlotsEmployee cvse ON cvss.AdminScheduleSlotsID = cvse.AdminScheduleSlotsID
LEFT JOIN ABCDocumentation cd ON cvse.AdminScheduleSlotsEmployeeID = cd.AdminScheduleSlotsEmployeeID
LEFT JOIN Employee e ON cvse.EmplID = e.EMPLID
OUTER APPLY
(
SELECT v.ColorCode, mfg.Description
FROM ABCDocumentation cd1
LEFT JOIN ABCDocumentation cd2ON cd1.ABCDocumentationID = cd2.PairID
LEFT JOIN Medicine v ON v.MedicineID = cd1.MedicineID
LEFT JOIN Manufacturers mfg ON v.MFG_Seq = mfg.MFG_Seq
WHERE cd2.ABCDocumentationID = dt.ABCDocumentationID
) AS Parent
WHERE TIME >= CAST(GETDATE() AS Date) AND TIME < CAST(DATEADD(day, 1, GETDATE()) AS DATE)
AND CampusID = '06'
AND cvse.AdminScheduleSlotsEmployeeID IS NOT NULL
ORDER BY TIME
please
declare #tm table (id int identity, timee time(7))
insert into #tm (timee) values ('01:05:45'),
('10:15:18'),
('14:18:59'),
('09:15:10'),
('18:19:21'),
('21:05:17')
this is a default
select * from #tm order by id
this is a, what do you need
select tm.*,
iif(tm.part_time = 1, 'morning', 'evening') m_e from (select
case
when timee between '09:00:00' and '19:00:00' then 1
else 2 end part_time,
*
from #tm) tm
order by part_time, timee

How to combine two select query results into one result using sp?

I'm declaring two comparison dates, I want data from both dates. I'm using left join for combined propose but this is not correct way. I'm missing some data. Instead of left join which one is best for?
result
productid FirstQty SecondQty FirstProductRevenue SecondProductRevenue
COCAK117 1 2 1370.00 1440.00
COCAK632 1 2 1125.00 2250.00
COCAK656 1 NULL 795.00 NULL
COCAK657 1 2 720.00 2090.00
COCAK775 3 1 2475.00 825.00
I'm getting data from full of first table and matching productid from second table, but I want total productid's from both the tables.
CREATE PROCEDURE [dbo].[Orders]
(
#StartDate DATETIME,
#EndDate DATETIME,
#StartDate1 DATETIME,
#EndDate1 DATETIME,
#Rowname VARCHAR(100),
#AssociateName VARCHAR(50)
)
--[Orders] '05/03/2015','05/03/2015','05/05/2015','05/07/2015','Most Gifted Cakes','all'
AS BEGIN
if(#AssociateName='all')
BEGIN
----First duration for all associates-----
select t1.productid,t1.FirstQty,t2.SecondQty,t1.FirstProductRevenue,t2.SecondProductRevenue
from
(select op.Productid
, count(op.ProductId)as FirstQty
, Round(Sum(op.Price*op.Quantity),0) as FirstProductRevenue
from Orderdetails od
inner join (select Distinct Orderid,productid,Price,Quantity from Orderproducts) op on op.Orderid=od.Orderid
inner JOIN City ct ON od.RecipientCityName = ct.CityName
INNER JOIN Associates ass ON Ct.AssociateId = ass.AssociateId
Inner join HomepageRowWiseProducts hr on op.ProductId=hr.Productid
where Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate and #EndDate
and (od.TransactionId IS NOT NULL or ltrim(od.TransactionId) != '')
and #Rowname=hr.HomepageRow_name and hr.status=1
Group by op.Productid
) t1
----Second duration for all associates-----
left join
(select op.Productid
, count(op.ProductId)as SecondQty
, Round(Sum(op.Price*op.Quantity),0) as SecondProductRevenue
from Orderdetails od
inner join (select Distinct Orderid,productid,Price,Quantity from Orderproducts) op on op.Orderid=od.Orderid
inner JOIN City ct ON od.RecipientCityName = ct.CityName
INNER JOIN Associates ass ON Ct.AssociateId = ass.AssociateId
Inner join HomepageRowWiseProducts hr on op.ProductId=hr.Productid
where Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate1 and #EndDate1
and (od.TransactionId IS NOT NULL or ltrim(od.TransactionId) != '')
and #Rowname=hr.HomepageRow_name and hr.status=1
Group by op.Productid
) t2 on t1.productid=t2.productid
END
You can try to get all data at once and then sum only date ranges that you want.
I could made some mistake here as I don't have your data structures.
However you should get the idea how you can implement it.
select op.Productid
, sum( case when Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate and #EndDate then
1 else 0 end) FirstQty
, sum( case when Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate1 and #EndDate1 then
1 else 0 end) SecondQty,
, Round(Sum( case when Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate and #EndDate
then op.Price*op.Quantity
else 0 end),0) as FirstProductRevenue
, Round(Sum( case when Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate1 and #EndDate1
then op.Price*op.Quantity
else 0 end),0) as SecondProductRevenue
from Orderdetails od
inner join (select Distinct Orderid,productid,Price,Quantity from Orderproducts) op on op.Orderid=od.Orderid
inner JOIN City ct ON od.RecipientCityName = ct.CityName
INNER JOIN Associates ass ON Ct.AssociateId = ass.AssociateId
Inner join HomepageRowWiseProducts hr on op.ProductId=hr.Productid
where ( Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate and #EndDate
Or Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate1 and #EndDate1 )
and (od.TransactionId IS NOT NULL or ltrim(od.TransactionId) != '')
and #Rowname=hr.HomepageRow_name and hr.status=1
Group by op.Productid
Maybe this (simplified):
select coalesce(t1.productid, t2.productid) as productid, t1.FirstQty, t2.SecondQty, t1.FirstProductRevenue, t2.SecondProductRevenue
from
(select ...) t1
----Second duration for all associates-----
full join
(select ...) t2 on t1.productid = t2.productid
try this
you create a function as follows
CREATE FUNCTION OrderDetails (
#StartDate DATETIME,
#EndDate DATETIME
)
RETURNS #tblList TABLE
(
orderId VARCHAR(1000)
)
AS
BEGIN
select orderid
insert into #tblList
from Orderdetails od inner JOIN City ct ON od.RecipientCityName = ct.CityName
INNER JOIN Associates ass ON Ct.AssociateId = ass.AssociateId
Inner join HomepageRowWiseProducts hr on op.ProductId=hr.Productid
and (od.TransactionId IS NOT NULL or ltrim(od.TransactionId) != '')
and #Rowname=hr.HomepageRow_name and hr.status=1
where Convert(datetime,Convert(Varchar(50),od.DeliveryDate,101)) between #StartDate and #EndDate
return
end
then call the function in your stored procedure according to the date you are passing you don't miss any records
CREATE PROCEDURE [dbo].[Orders]
(
#StartDate DATETIME,
#EndDate DATETIME,
#StartDate1 DATETIME,
#EndDate1 DATETIME,
#Rowname VARCHAR(100),
#AssociateName VARCHAR(50)
)
--[Orders] '05/03/2015','05/03/2015','05/05/2015','05/07/2015','Most Gifted Cakes','all'
AS BEGIN
if(#AssociateName='all')
BEGIN
----First duration for all associates-----
select op1.Productid
, count(op.ProductId)as FirstQty
,count(op1.ProductId)as SecondQty
, Round(Sum(op.Price*op.Quantity),0) as FirstProductRevenue
, Round(Sum(op1.Price*op1.Quantity),0) as FirstProductRevenue
from (select Distinct Orderid,productid,Price,Quantity from Orderproducts opp inner join [Your_ScemaName].[OrderDetails](#StartDate,#EndDate) od on opp.Orderid=od.Orderid ) op
inner join (select Distinct Orderid,productid,Price,Quantity from Orderproducts opp inner join [Your_ScemaName].[OrderDetails](#StartDate1,#EndDate1) od on opp.Orderid=od.Orderid ) op1
-- since 1=1 is always true you will get all data
on 1=1
Group by op.Productid
end
end

What is an alternative to using subqueries in this SQL query that will give us better performance?

We have a Transact-SQL query that contains subqueries in the where section. Inner Join gives unwanted results of adding tOrderDetails fields multiple times so we resorted to the (poor performance) sub queries. It is timing out often. We need the same result but with better performance. Any suggestions?
SELECT sum (cast(Replace(tOrders.TotalCharges,'$','') as money)) as TotalCharges
FROM [ArtistShare].[dbo].tOrders
WHERE tOrders.isGiftCardRedemption = 0
and tOrders.isTestOrder=0
and tOrders.LastDateUpdate between #startDate and #endDate
and (SELECT count(tORderDetails.ID)
from tORderDetails
where tORderDetails.ORderID = tORders.ORderID
and tOrderDetails.isPromo=1) = 0
and (SELECT top 1 tProjects.ProjectReleaseDt
from tProjects JOIN tOrderDetails
on tOrderDetails.ProjectID = tProjects.ID
where tOrderDetails.OrderID = tOrders.OrderID) >= #startDate
Why do you count records if you just want to know if at least one exists or not exists?
SELECT Sum (Cast(Replace(tOrders.TotalCharges, '$', '') AS MONEY)) AS TotalCharges
FROM [ArtistShare].[dbo].tOrders
WHERE tOrders.isGiftCardRedemption = 0
AND tOrders.isTestOrder = 0
AND tOrders.LastDateUpdate BETWEEN #startDate AND #endDate
AND NOT EXISTS
(
SELECT 1 FROM tORderDetails
WHERE tORderDetails.ORderID = tORders.ORderID
AND tOrderDetails.isPromo = 1 )
AND EXISTS
(
SELECT 1 FROM tProjects INNER JOIN tOrderDetails
ON tOrderDetails.ProjectID = tProjects.ID
WHERE tOrderDetails.OrderID = tOrders.OrderID
AND tProjects.ProjectReleaseDt >= #startDate )
Although I think EXISTS may have the best performance, you might also consider the following approach. When you mention in your question that your query has "unwanted results of adding tOrderDetails fields multiple times" it's probably because you have multiple tOrderDetail records so you need to collapse them with a GROUP BY. Rather than using a correlated sub-query which is very inefficient, use a single sub-query with INNER JOIN like this.
SELECT
sum (cast(Replace(tOrders.TotalCharges, '$', '') as money)) as TotalCharges
FROM [ArtistShare].[dbo].tOrders
INNER JOIN (
SELECT OrderID
FROM tOrderDetails d INNER JOIN tProjects p on d.ProjectID = p.ID
WHERE d.isPromo = 0 AND p.ProjectReleaseDt > #startDate
GROUP BY OrderID
) qualifyingOrders ON qualifyingOrders.OrderID = tOrders.OrderID
WHERE tOrders.isGiftCardRedemption = 0
and tOrders.isTestOrder=0
and tOrders.LastDateUpdate between #startDate and #endDate
Again, you should compare this with the EXISTS approach to see which one performs better and makes the most sense for what you are trying to achieve.
One additional approach you could try is to put your sub-query data in to temporary tables first and then query on those.
Obviously I don't have your data or understand the data relationships exactly but hopefully the example below will get you on your way to the correct query for your needs.
Worth comparing the performance at least.
DECLARE #ORDERDETAILS TABLE
(
INT idCount,
INT orderId
)
INSERT INTO
#ORDERDETAILS
SELECT
COUNT(tORderDetails.ID),
tORderDetails.ORderID
WHERE
tOrderDetails.isPromo = 1
GROUP BY
tORderDetails.ORderID
DECLARE #PROJECTS TABLE
(
DATETIME releaseDte,
INT orderId
)
INSERT INTO
#PROJECTS
SELECT
tProjects.ProjectReleaseDt,
tOrderDetails.OrderID
FROM
tProjects JOIN tOrderDetails ON tOrderDetails.ProjectID = tProjects.ID
SELECT
sum (cast(Replace(tOrders.TotalCharges,'$','') as money)) as TotalCharges
FROM
[ArtistShare].[dbo].tOrders JOIN #ORDERDETAILS ON #ORDERDETAILS.orderId = tORders.ORderID
JOIN #PROJECTS ON #PROJECTS.orderId = tOrders.OrderID)
WHERE tOrders.isGiftCardRedemption = 0
and tOrders.isTestOrder=0
and tOrders.LastDateUpdate between #startDate and #endDate
AND #ORDERDETAILS.idCount = 0
--and (SELECT count(tORderDetails.ID)
-- from tORderDetails
-- where tORderDetails.ORderID = tORders.ORderID
-- and tOrderDetails.isPromo=1) = 0
AND #PROJECTS.releaseDte >= #startDate --NOT QUITE SURE WHAT THIS EXACTLY MIMICS WHAT YOU ARE TRYING TO DO BUT IF NOT YOU COULD REMOVE THE JOIN AND SUB-QUERY AS FOLLOWS
--AND (SELECT TOP 1 #PROJECTS.releaseDte WHERE PROJECTS.orderId = tOrders.OrderID) >= #startDate
--and (SELECT top 1 tProjects.ProjectReleaseDt
-- from tProjects JOIN tOrderDetails
-- on tOrderDetails.ProjectID = tProjects.ID
-- where tOrderDetails.OrderID = tOrders.OrderID) >= #startDate

Filtering ON sub query only

I have a query that returns four columns and i want to apply a date range/filter to the two of the columns only , Previously i wrote My query as below.
SELECT TOP (100) PERCENT dbo.Collector.name,
dbo.Debtor.batchno AS batchno,
COUNT(dbo.Debtor.id) AS cases,
SUM(dbo.Debtor.totalDebt) AS amount,
SUM(dbo.Payment.payment) AS collection,
SUM(dbo.Debtor.balance) AS OSBalance,
dbo.debtor.receivedDate
FROM dbo.Payment WITH (NOLOCK)
INNER JOIN dbo.Collector WITH (NOLOCK)
ON dbo.Payment.collectorid = dbo.Collector.id
INNER JOIN dbo.Debtor WITH (NOLOCK)
ON dbo.Payment.debtorid = dbo.Debtor.id
INNER JOIN dbo.Client WITH (NOLOCK)
ON dbo.debtor.clientid = dbo.Client.id
WHERE dbo.Payment.paymentdate BETWEEN ' 2011-02-27 00:00:00.000 '
AND '2014-02-27 23:59:59.000'
AND dbo.Payment.confirmed='Y'
GROUP BY
dbo.Collector.name,
dbo.Debtor.batchno,
dbo.debtor.receivedDate
ORDER BY
dbo.debtor.receivedDate DESC
The above filter all the columns by the paymentdate. However i want the filter by payment date to apply to only two columns namely cases and collecton like below.
SELECT TOP (100) PERCENT dbo.Collector.name,
dbo.Debtor.batchno AS batchno,
(
SELECT COUNT(dbo.Debtor.id)
FROM dbo.Debtor d INNER JOIN dbo.Payment p ON d.id = p.debtorid
WHERE paymentdate BETWEEN ' 2011-02-27 00:00:00.000 '
AND '2014-02-27 23:59:59.000'
AND batchno=dbo.Debtor.batchno
AND dbo.collector.name= dbo.collector.name
) AS cases,
SUM(dbo.Debtor.totalDebt) AS amount,
(
SELECT SUM(dbo.Payment.payment)
FROM dbo.Payment
WHERE paymentdate BETWEEN ' 2011-02-27 00:00:00.000 '
AND '2014-02-27 23:59:59.000'
AND batchno= dbo.Debtor.batchno
AND name= dbo.collector.name
) AS collection,
SUM(dbo.Debtor.balance) AS OSBalance,
dbo.debtor.receivedDate
FROM dbo.Payment WITH (NOLOCK)
INNER JOIN dbo.Collector WITH (NOLOCK)
ON dbo.Payment.collectorid = dbo.Collector.id
INNER JOIN dbo.Debtor WITH (NOLOCK)
ON dbo.Payment.debtorid = dbo.Debtor.id
INNER JOIN dbo.Client WITH (NOLOCK)
ON dbo.debtor.clientid = dbo.Client.id
WHERE dbo.Payment.confirmed='Y'
GROUP BY
dbo.Collector.name,
dbo.Debtor.batchno,
dbo.debtor.receivedDate
ORDER BY
dbo.debtor.receivedDate DESC
The above doesnt work for me. Also i want to group by only batchno and name. Plus the collection and cases columns must filter by payment date . Pls how do i achieve this ? Any help would be appreciated. warmest regard

counting items that match within select statement

We have a stored procedure that executes against some fairly large tables and while joining to a larger table it is also keeping a tally of how many records match the corresponding batch_id. What I am trying to figure out is can I improve this with a function for the count or some other means? Trying to get rid of the nested SELECT COUNT(*) statement. The CCTransactions table is 1.4 million rows and the BatchItems is 6.6 million rows.
SELECT a.ItemAuthID, a.FeeAuthID, a.Batch_ID, a.ItemAuthCode,
a.FeeAuthCode, b.Amount, b.Fee,
(SELECT COUNT(*) FROM BatchItems WHERE Batch_ID = a.Batch_ID) AS BatchCount,
ItemBillDate, FeeBillDate, b.AccountNumber,
b.Itemcode, ItemAuthToken, FeeAuthToken,
cc.ItemMerchant, cc.FeeMerchant
FROM CCTransactions a WITH(NOLOCK)
INNER JOIN BatchItems b WITH(NOLOCK)
ON a.Batch_ID = b.Batch_ID
INNER JOIN CCConfig cc WITH(NOLOCK)
ON a.ClientCode = cc.ClientCode
WHERE ((ItemAuthCode > '' AND ItemBillDate IS NULL)
OR (FeeAuthCode > '' AND FeeBillDate IS NULL))
AND TransactionDate BETWEEN DATEADD(d,-7,GETDATE())
AND convert(char(20),getdate(),101) + ' ' + #Cutoff
ORDER BY TransactionDate
When your DBMS supports WIndowed Aggregate Functions you can rewrite it to
COUNT(*) OVER (PARTITION BY Batch_ID)
Of course this only returns the number of rows per Batch_ID returned by the SELECT. if the inner join results in less rows, it's not the correct number.
Then it might be more efficient (depending on your DBMS) to rewrite the Scalar Subquery to a join:
SELECT a.ItemAuthID, a.FeeAuthID, a.Batch_ID, a.ItemAuthCode,
a.FeeAuthCode, b.Amount, b.Fee,
dt.BatchCount,
ItemBillDate, FeeBillDate, b.AccountNumber,
b.Itemcode, ItemAuthToken, FeeAuthToken,
cc.ItemMerchant, cc.FeeMerchant
FROM CCTransactions a WITH(NOLOCK)
INNER JOIN BatchItems b WITH(NOLOCK)
ON a.Batch_ID = b.Batch_ID
INNER JOIN CCConfig cc WITH(NOLOCK)
ON a.ClientCode = cc.ClientCode
INNER JOIN
(
SELECT BatchCount, COUNT(*) AS BatchCount
FROM BatchItems
GROUP BY Batch_ID
) AS dt ON a.Batch_ID = dt.Batch_ID
WHERE ((ItemAuthCode > '' AND ItemBillDate IS NULL)
OR (FeeAuthCode > '' AND FeeBillDate IS NULL))
AND TransactionDate BETWEEN DATEADD(d,-7,GETDATE())
AND convert(CHAR(20),getdate(),101) + ' ' + #Cutoff
ORDER BY TransactionDate