How to optimize this SQL query summing amounts across related tables? - sql

Question

I have an Access project that implements SQL, and I have been working on optimizing a reconciliation process. This process uses a voucher system which links all tables together.
Each record in each table has a specific voucher ID in which an amount is associated with.
The vouchers themselves are unique and can contain multiple voucher numbers, which can be seen below.
Table: Rec_Vouchers
v_id v_num voucher
1 12341234 12341234
2 10101010 10101010;22222222
2 22222222 10101010;22222222
...
I have 8 other tables that are linked by these voucher ID's. I'm trying to join all of the tables together to show the distict voucher ID and voucher and all corresponding sums of amounts for each table with that specific voucher ID. Below is the query and a sample of the results. I've worked on this for a while now, and it's starting to give me a headache. This query works, but takes way to long to execute.
Also, at some point, I need to match all of these values together to determine if a voucher is "Matching", "Not matching", or "Matching with a difference". So far I've only tried creating a function within the below code that would return a string value of "M", "NM", or "MwD" to display in the column for each voucher. Again, this works, but takes an extremely long time. I've also tried letting VBA do the dirty work with the query's returned recordset, and this takes a good amount of time too, but not as long as creating the function within my sql query. This is the next step, so if you could help with all of this that would be great, but I really just need to optimize the query I have given.
I know this is a lot to wrap your head around, so let me know if you need any more information. Any help would be appreciated. Thanks!
select a.v_id, a.voucher,
(Select sum(b.amount) from rec_month_4349_test b where b.voucher = a.v_id) as GL,
(Select sum(c.payments) from rec_daily_balancing_test c where c.voucher = a.v_id) as DB,
(Select count(x.v_num) from rec_vouchers x where a.v_id = x.v_id and x.v_num not like 'ONL%') as GLcount,
(select count(c.batch_num) from rec_daily_balancing_test c where a.v_id = c.voucher) as DBcount,
(select sum(d.amount) from rec_ed_test d where a.v_id = d.voucher) as ED,
(select sum(e.batchtotal) from rec_eft_batches_new_test e where a.v_id = e.voucher) as EFT,
(select sum(f.batchtotal) from rec_check_batch_test f where a.v_id = f.voucher) as CHK,
(select sum(g.idxtotal) from rec_lockbox_test g where a.v_id = g.voucher) as LBX,
(select sum(h.amount) from rec_lcdl_test h where a.v_id = h.voucher) as LCDL,
((select sum(i.payment_amount) from rec_electronic_files_test i where a.v_id = i.voucher) + (select sum(j.amount) from rec_electronic_edits_test j where a.v_id = j.voucher)) as Elec
from rec_vouchers a
group by a.v_id, a.voucher
Sample Results:
v_id GL DB GLcount DBcount ED EFT CHK LBX LCDL Elec
6131 19204.00 19204.00 1 1 NULL NULL NULL NULL NULL NULL
6132 125330.00 14932.00 6 6 NULL NULL NULL NULL NULL 14932.00
6133 18245.00 NULL 2 0 NULL NULL NULL NULL NULL NULL
6175 98.93 98.93 1 1 NULL 98.93 NULL NULL NULL NULL

Answer 1

It is tempting to say that the "traditional" way to write this query is by moving the tables to the from clause using join predicates. However, that would probably introduce unnecessary cartesian products. Your method is actually ok; the alternative would be doing left joins to aggregated subqueries.
The killer on performance is probably due to cycling through the tables to find the matches. You can significantly improve performance by having an index on the fields used in the where clause for each query. For the first two tables, for instance, you should have an index on rec_month_4349_test(voucher) and rec_daily_balancing_test(voucher).
In SQL Server, you can further optimize this query by including the variable used for summation in the index as well. The following indexes would be better: rec_month_4349_test(voucher, amount) and rec_daily_balancing_test(voucher, payments) (or you can include them in the index without being searchable, which is a bit more advanced).
This optimization works in most databases (an index-scan rather than an index-lookup). I don't know if it works in MS Access (a software product that I try to avoid when possible).
Remember, you would need to do this for all the tables.

Answer 2

Not sure if this is the best solution, but I created separate views for each table to select the voucher and the sum of the amounts for each specific voucher. Each view looked similair to the following:
rec_sum4349
SELECT voucher, sum(amount) AS GL
FROM rec_month_4349_test
GROUP BY voucher
I then have one view that combines all of the separate view together using full joins like the following:
rec_vouch_test
SELECT a.voucher, a.GL, b.DB
FROM rec_sum4349 a
FULL JOIN rec_sumDB b
ON a.voucher = b.voucher
WHERE a.voucher IS NOT NULL AND a.voucher <> ''
ORDER BY a.voucher
After I saw that this worked really well, I created the views for the rest of the tables that I needed summed amounts for and added them to the above view. The results are exactly what I was looking for and the run-time was cut down from almost 2 minutes to the matter of a couple seconds! Thanks for all the help. Now on to matching everything up!

Answer 3

correlated sub queries is the last choice i would prefer.
i would suggest to write sub-query and join each table so that each table can utilize the indexes on them.
Create the following indexes on each table and see the below query.
rec_vouchers
Clustered Index (v_id , voucher)
Filtered Non Clustered Index(v_num) WHERE v_num NOT LIKE 'ONL%'
rec_month_4349_test
Non Clustered Index(voucher) Include (amount)
rec_daily_balancing_test
Non Clustered Index(voucher) Include (payments)
rec_ed_test
Non Clustered Index(voucher) Include (amount)
rec_eft_batches_new_test
Non Clustered Index(voucher) Include (batchtotal)
rec_check_batch_test
Non Clustered Index(voucher) Include (batchtotal)
rec_lockbox_test
Non Clustered Index(voucher) Include (idxtotal)
rec_lcdl_test
Non Clustered Index(voucher) Include (amount)
rec_electronic_files_test
Non Clustered Index(voucher) Include (payment_amount)
rec_electronic_edits_test
Non Clustered Index(voucher) Include (amount)
SELECT a.v_id,a.voucher
,t1.GL ,t2.DB ,t3.GLcount
,t4.DBcount ,t5.ED ,t6.EFT
,t7.CHK ,t8.LBX ,t9.LCDL
,(t10.Elec1+t11.Elec2) AS Elec
FROM
( SELECT t0.v_id ,t0.voucher
FROM rec_vouchers t0
GROUP BY t0.v_id ,t0.voucher
)a
JOIN
( SELECT SUM(b.amount) AS GL,b.voucher
FROM rec_month_4349_test b
Group By b.voucher
) t1
ON a.v_id=t1.voucher
JOIN
( SELECT SUM(c.payments) AS DB,c.voucher
FROM rec_daily_balancing_test c
Group By c.voucher
) t2
ON a.v_id=t2.voucher
JOIN
( SELECT COUNT(x.v_num) AS GLcount,x.v_id
FROM rec_vouchers x
WHERE x.v_num NOT LIKE 'ONL%'
Group BY x.v_id
) t3
ON a.v_id=t3.v_id
JOIN
( SELECT COUNT(c.batch_num) AS DBcount,c.voucher
FROM rec_daily_balancing_test c
Group By c.voucher
) t4
ON a.v_id=t4.voucher
JOIN
( SELECT SUM(d.amount) AS ED,d.voucher
FROM rec_ed_test d
Group By d.voucher
) t5
ON a.v_id=t5.voucher
JOIN
( SELECT SUM(e.batchtotal) AS EFT,e.voucher
FROM rec_eft_batches_new_test e
Group By e.voucher
) t6
ON a.v_id=t6.voucher
JOIN
( SELECT SUM(f.batchtotal) AS CHK,f.voucher
FROM rec_check_batch_test f
Group By f.voucher
) t7
ON a.v_id=t7.voucher
JOIN
( SELECT SUM(g.idxtotal) AS LBX,g.voucher
FROM rec_lockbox_test g
Group By g.voucher
) t8
ON a.v_id=t8.voucher
JOIN
( SELECT SUM(h.amount) AS LCDL,h.voucher
FROM rec_lcdl_test h
Group By h.voucher
) t9
ON a.v_id=t9.voucher
JOIN
( SELECT SUM(i.payment_amount) AS Elec1,i.voucher
FROM rec_electronic_files_test i
GROUP BY i.voucher
) t10
ON a.v_id=t10.voucher
JOIN
( SELECT SUM(j.amount) AS Elec2,j.voucher
FROM rec_electronic_edits_test j
GROUP BY j.voucher
) t11
ON a.v_id=t11.voucher