How would I translate this SQL query into a Raven Map/Reduce query? - ravendb

Question

Following on from my previous question at When is a groupby query evaluated in RavenDB? I decided to completely restructure the data into a format that is theoretically easier to query on.
Having now created the new data structure, I am struggling to find how to query it.
It took me 30 seconds to write the following SQL query which gives me exactly the results I need:
SELECT GroupCompanyId, AccountCurrency, AccountName, DATEPART(year, Date) AS Year,
(SELECT SUM(Debit) AS Expr1
FROM Transactions AS T2
WHERE (T1.GroupCompanyId = GroupCompanyId) AND (T1.AccountCurrency = AccountCurrency) AND (T1.AccountName = AccountName) AND (DATEPART(year,
Date) < DATEPART(year, T1.Date))) AS OpeningDebits,
(SELECT SUM(Credit) AS Expr1
FROM Transactions AS T2
WHERE (T1.GroupCompanyId = GroupCompanyId) AND (T1.AccountCurrency = AccountCurrency) AND (T1.AccountName = AccountName) AND (DATEPART(year,
Date) < DATEPART(year, T1.Date))) AS OpeningCredits, SUM(Debit) AS Db, SUM(Credit) AS Cr
FROM Transactions AS T1
WHERE (DATEPART(year, Date) = 2011)
GROUP BY GroupCompanyId, AccountCurrency, AccountName, DATEPART(year, Date)
ORDER BY GroupCompanyId, AccountCurrency, Year, AccountName
So far I have got the Map/Reduce as follows, which from Studio appears to give the correct results - i.e. it breaks down and groups the data by date.
public Transactions_ByDailyBalance()
{
Map = transactions => from transaction in transactions
select new
{
transaction.GroupCompanyId,
transaction.AccountCurrency,
transaction.Account.Category,
transaction.Account.GroupType,
transaction.AccountId,
transaction.AccountName,
transaction.Date,
transaction.Debit,
transaction.Credit,
};
Reduce = results => from result in results
group result by new
{
result.GroupCompanyId,
result.AccountCurrency,
result.Category,
result.GroupType,
result.AccountId,
result.AccountName,
result.Date,
}
into g
select new
{
GroupCompanyId = g.Select(x=>x.GroupCompanyId).FirstOrDefault(),
AccountCurrency = g.Select(x=>x.AccountCurrency).FirstOrDefault(),
Category=g.Select(x=>x.Category).FirstOrDefault(),
GroupType=g.Select(x=>x.GroupType).FirstOrDefault(),
AccountId = g.Select(x=>x.AccountId).FirstOrDefault(),
AccountName=g.Select(x=>x.AccountName).FirstOrDefault(),
Date=g.Select(x=>x.Date).FirstOrDefault(),
Debit=g.Sum(x=>x.Debit),
Credit=g.Sum(x=>x.Credit)
};
Index(x=>x.GroupCompanyId,FieldIndexing.Analyzed);
Index(x=>x.AccountCurrency,FieldIndexing.Analyzed);
Index(x=>x.Category,FieldIndexing.Analyzed);
Index(x=>x.AccountId,FieldIndexing.Analyzed);
Index(x=>x.AccountName,FieldIndexing.Analyzed);
Index(x=>x.Date,FieldIndexing.Analyzed);
}
}
However, I can't work out how to query the data at one go.
I need the opening balance as well as the period balance, so I ended up writing this query which takes as a parameter the account. Following on from Oren's comments to my previous question, that I was mixing Linq with Lucene query, having rewritten the query, I've basically ended up again with a mixed query.
Even though I am showing in the SQL query above that I am filtering by year, in fact I need to be able to determine the current balance from any day.
private LedgerBalanceDto GetAccountBalance(BaseAccountCode account, DateTime periodFrom, DateTime periodTo, string queryName)
{
using (var session = MvcApplication.RavenSession)
{
var query = session.Query<Transactions_ByDailyBalance.Result, Transactions_ByDailyBalance>()
.Where(c=>c.AccountId==account.Id && c.Date>=periodFrom && c.Date<=periodTo)
.OrderBy(c=>c.Date)
.ToList();
var debits = query.Sum(c => c.Debit);
var credits = query.Sum(c => c.Credit);
var ledgerBalanceDto = new LedgerBalanceDto
{
Account = account,
Credits = credits,
Debits = debits,
Currency = account.Currency,
CurrencySymbol = account.CurrencySymbol,
Name = queryName,
PeriodFrom = periodFrom,
PeriodTo = periodTo
};
return ledgerBalanceDto;
}
}
Required result:
GroupCompanyId AccountCurrency AccountName Year OpeningDebits OpeningCredits Db Cr
Groupcompanies-2 EUR Customer 1 2011 148584.2393 125869.91 10297.6891 28023.98
Groupcompanies-2 EUR Customer 2 2011 236818.0054 233671.55 50959.85 54323.38
Groupcompanies-2 USD Customer 3 2011 69426.11761 23516.3776 10626.75 0
Groupcompanies-2 USD Customer 4 2011 530587.9223 474960.51 97463.544 131497.16
Groupcompanies-2 USD Customer 5 2011 29542.391 28850.19 4023.688 4231.388
Any suggestions would be greatly appreciated
Jeremy
In answer to the comment
I basically ended up doing pretty much the same thing. Actually, I wrote an index that does it in only two hits - once for the opening balance and again for the period balance. This is almost instantaneous for grouping by the account name, category etc.
However my problem now is getting a daily running balance for the individual account. If I bring down all the data for the account and the period, its not a problem - I can sum the balance on the client, however, when the data is paged, and the debits and credits are grouped by Date and Id, the paging cuts across the date, so the opening/closing balance is not correct.
Page 1
Opening balance until 26/7/12 = 0
25/7/12 Acct1 Db 100 Cr 0 Bal +100 Runn Bal +100
26/7/12 Acct1 Db 100 Cr 0 Bal +100 Runn Bal +200
26/7/12 Acct1 Db 200 Cr 0 Bal +200 Runn Bal +400
Closing balance until 26/7/12 = +400
Page 2
Opening balance until 26/7/12 = +450 (this is wrong - it should be the balance at the end of Page 1, but it is the balance until the 26/7/12 - i.e. includes the first item on Page 2)
26/7/12 Acct1 Db 50 Cr 0 Bal +50 Runn Bal +500 (should be +450)
27/7/12 Acct1 Db 60 Cr 0 Bal +60 Runn Bal +560 (should be +510)
I just can't think up an algorithm to handle this.
Any ideas?

Answer 1

Hi this is a problem I have also faced recently with RavenDb when I needed to retrieve rolling balances as at any imaginable date. I never found a way of doing this all in one go but I managed reduce the amount of documents that I needed to pull back in order to calculate the rolling balance.
I did this by writing multiple map reduce indexes that summed up the value of transactions within specific periods:
My First Summed up the value of all transactions grouped at the year level
My Second index Summed up the value of all transactions at the Day level
So if someone wanted their account balance As At 1st June 2012 I would:
Use the Year level Map-reduce index to get the Value of transactions for years up to 2012 and summed them together (so if transactions started being captured in 2009 I should be pulling back 3 documents)
Use the Day level Map-reduce index to get all documents from the start of the year and the 1st of June
I then Added the Day totals to the year totals for my final rolling balance (I could have also had a monthly map reduce as well but didn't bother).
Anyway not as quick as in SQL but it was the best alternative I could come up with to avoid bringing back every single transaction