Sum function in mule dataweave takes too much time - sum

Question

I have one ArrayList which contains more than 45,000 records. I want to create a new List which will contain Sum of all values when combination of ID and Name value present in the list. e.g. Below is my input List. I am creating this list by reading xls file and storing it in one variable called "myList" -
[
{ID=481, name =ABCD, value=100},
{ID=481, name =ABCD, value=50},
{ID=2053, name =XYZ, value=300}
]
My Code -
%dw 1.0
%output application/java
%function getIdname(ID, name) (ID ++ " - " ++ name)
%function addAllValues(ID, name) sum ((flowVars.myList.Data filter (getIdname($.ID,$.name) == getIdname(ID, name))).value)
---
{
Data: flowVars.myList.Data map ((payload01 , indexOfPayload01) -> {
ID: payload01.ID,
name: payload01.name,
finaValue : addAllValues(payload01.ID, payload01.name)
})
}
Output -
Data=[
{ID=481, name =ABCD, finalValue=150},
{ID=2053, name =XYZ, finalValue=300}
]
Here, I am getting desired output as above for a file with 5 or 10 records. But, if I am using actual file with more than 45,000 records it is taking too much time to execute my code and it is not creating any output. Also, I am not getting any exception. Can anyone suggest what I am doing wrong here. Why it is taking so much time to sum equal records. I have checked it for 40 mins, but still not received any output

Answer 1

You are filtering 45000 records in every single iteration of the map which is causing this delay (i.e. you are filtering 45000 times). You have to filter/group only once. Also your dataweave will not produce the output you want because there is no distinctBy used.
Instead try this:
%dw 1.0
%output application/java
%var dataByIdName = flowVars.myList.Data groupBy ($.ID ++ $.name)
---
{
Data: dataByIdName map {
ID: $[0].ID,
name: $[0].name,
finalValue: sum $.*value
}
}
This doesn't need distinctBy and also you only group once.