Mule DB data retrieval into chunks - mule

Question

We are trying to extract approx. 40 GB data from database and want to generate multiple csv files. We used mule DB connector in streaming fashion, which is returning 'ResultSetIterator'
Q1) How to convert this ResultSetIterator to arraylist? or any readable format which we can use further to generate files
Q2) We tried using For-Each component to split this data in chunks, its working for limited set of data and for huge data giving SerializationException
In below input snippet we are making chunks of data using for-each and providing it to batch process for multiple files
<batch:job name="testBatchWithDBOutside">
<batch:input>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
</batch:input>
<batch:process-records>
<batch:step name="Batch_Step">
<batch:commit size="10" doc:name="Batch Commit">
<object-to-string-transformer doc:name="Object to String"/>
<logger message="#[payload]" level="INFO" doc:name="Logger"/>
<file:outbound-endpoint path="C:\output" outputPattern="#[message.id].txt" responseTimeout="10000" doc:name="File"/>
</batch:commit>
</batch:step>
</batch:process-records>
</batch:job>
<flow name="testBatchWithDBOutsideFlow" processingStrategy="synchronous">
<file:inbound-endpoint path="C:\input" responseTimeout="10000" doc:name="File"/>
<db:select config-ref="MySQL_Configuration" streaming="true" fetchSize="10" doc:name="Database">
<db:parameterized-query><![CDATA[select * from classicmodels]]></db:parameterized-query>
</db:select>
<foreach batchSize="5" doc:name="For Each">
<batch:execute name="testBatchWithDBOutside" doc:name="testBatchWithDBOutside"/>
</foreach>
</flow>

Answer 1

Q1. You don't want to convert the Iterator to a List, as this will defeat the purpose of streaming from the DB connector and load all records into memory. Mule handles Iterators and Lists in the same way anyway.
Q2. The batch module implies a for-each operation. The output of batch:input needs to be a List or an Iterator. You should be able to simplify this
<batch:job name="testBatch">
<batch:input>
<db:select config-ref="MySQL_Configuration" streaming="true" fetchSize="10" doc:name="Database">
<db:parameterized-query><![CDATA[select * from classicmodels]]></db:parameterized-query>
</db:select>
</batch:input>
<batch:process-records>
<batch:step name="Batch_Step">
<object-to-string-transformer doc:name="Object to String"/>
<file:outbound-endpoint path="C:\output" outputPattern="#[message.id].txt" responseTimeout="10000" doc:name="File"/>
</batch:step>
</batch:process-records>
</batch:job>
You will also need to replace the object-to-string-transformer with a component that converts a database record (the payload at this point will be a map where the key is the column name, and the value is the record value) into a csv line.
You can find a decent example in the Mule blog here: https://blogs.mulesoft.com/dev/anypoint-platform-dev/batch-module-reloaded/
Another option would be to remove the batch processor and use DataWeave to generate csv output and stream it to the file. This might be helpful: https://docs.mulesoft.com/mule-user-guide/v/3.7/dataweave-streaming
Dataweave will call next on the ResultSetIterator as it processes each record, and that Iterator will handle selecting chunks of records from the underlying database, so there is no queueing in between steps, or loading the full dataset into memory.
<flow name="batchtestFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/batch" allowedMethods="GET" doc:name="HTTP"/>
<db:select config-ref="Generic_Database_Configuration" streaming="true" doc:name="Database">
<db:parameterized-query><![CDATA[select * from Employees]]></db:parameterized-query>
</db:select>
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%input payload application/java
%output application/csv streaming=true, header=true, quoteValues=true
---
payload map ((e, i) -> {
surname: e.SURNAME,
firstname: e.FIRST_NAME
})]]></dw:set-payload>
</dw:transform-message>
<file:outbound-endpoint path="C:/tmp" outputPattern="testbatchfile.csv" connector-ref="File" responseTimeout="10000" doc:name="File"/>
</flow>

Answer 2

You want to use OutputHander. Make sure you have streaming turned on and then use a script component, for instance select groovy and handle each row one at a time like so:
// script.groovy
return {evt, out ->
payload.each { row ->
out << row.SOMECOLUMN.... }
} as OutputHandler
And the component in your xml
<scripting:transformer returnClass="TODO" doc:name="ScriptComponent">
<scripting:script engine="Groovy" file="script.groovy" />
</scripting:transformer>
If you want to return some output. However if you want to write to a file in your case you wouldn't use the variable out but instead write to your files.

Answer 3

I found a simple and quickest way as below:
Here DB connector is in streaming mode, and For-Each is splitting the records in given Batch Size
<flow name="testFlow" processingStrategy="synchronous">
<composite-source doc:name="Composite Source">
<quartz:inbound-endpoint jobName="test" cronExpression="0 48 13 1/1 * ? *" repeatInterval="0" connector-ref="Quartz" responseTimeout="10000" doc:name="Quartz">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<http:listener config-ref="HTTP_Listener_Configuration" path="/hit" doc:name="HTTP"/>
</composite-source>
<db:select config-ref="MySQL_Configuration" streaming="true" fetchSize="10000" doc:name="Database">
<db:parameterized-query><![CDATA[SELECT * FROM tblName]]></db:parameterized-query>
</db:select>
<foreach batchSize="10000" doc:name="For Each">
<dw:transform-message doc:name="Transform Message">
<dw:set-payload><![CDATA[%dw 1.0
%output application/csv
---
payload map {
field1:$.InterfaceId,
field2:$.Component
}]]></dw:set-payload>
</dw:transform-message>
<file:outbound-endpoint path="F:\output" outputPattern="#[message.id].csv" responseTimeout="10000" doc:name="File"/>
</foreach>
<set-payload value="*** Success ***" doc:name="Set Payload"/>
</flow>