Speeding up transforming DB select to XML with Mule 3.5 - mule

Question

I'm trying to convert data from a database query to XML. There are about 20,000 rows. It takes something like ten minutes to run, which seems unreasonably slow to me.
I noticed that eventContext has a getOutputStream method, but that throws an exception when I try to call it returns null whenever I call it. I also don't have the Java expertise to know the best way to stream the XML, though I'm sure I can look that up, I'd appreciate any hints there.
The structure of the XML I want is something like:
<myElements>
<myElement>
<someId>123454</someId>
<someProperty>Some Value</someProperty>
<mySubElements>
<mySubElement>
<anotherId>67890</anotherId>
<anotherProperty>Another Value</anotherProperty>
</mySubElement>
<mySubElement>
<anotherId>24680</anotherId>
<anotherProperty>Yet Another Value</anotherProperty>
</mySubElement>
</mySubElements>
</myElement>
</myElements>
The output of my query is something like this:
SOME_ID SOME_PROPERTY ANOTHER_ID ANOTHER_PROPERTY
12345 Some Value 67890 Another Value
12345 Some Value 24680 Yet Another Value
I'm using the component transformer, and my code there looks something like this:
public Object onCall(MuleEventContext eventContext) throws Exception {
List queryResults = (List) eventContext.getMessage().getPayload();
DocumentBuilderFactory docBuilderFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = docBuilderFactory.newDocumentBuilder();
// root elements
Document doc = docBuilder.newDocument();
Element myElements = doc.createElement("myElements");
doc.appendChild(myElements);
Element myElement = null;
Integer currentSomeId = null;
for (Integer i = 0; i < queryResults.size(); i++)
{
Map queryRow = (Map) queryResults.get(i);
if (!queryRow.get("SOME_ID").equals(currentSomeId))
{
myElement = doc.createElement("myElement");
/* populate my element, including with an empty mySubElements */
}
Element mySubElement = doc.createElement("myElement");
/* populate mySubElement */
}
ByteArrayOutputStream baos = new ByteArrayOutputStream();
TransformerFactory tf = TransformerFactory.newInstance();
Transformer t = tf.newTransformer();
t.setOutputProperty(OutputKeys.INDENT, "yes");
t.transform(new DOMSource(doc), new StreamResult(baos));
eventContext.getMessage().setPayload(doc);
return baos.toByteArray();
}
My mule flow looks like this:
<flow name="MyElementsFlow-Https">
<http:listener config-ref="HTTP_Listener_Configuration" path="${webapi.myelements.path}" allowedMethods="GET" doc:name="HTTP"/>
<flow-ref name="MyElementsFlow" doc:name="Flow Reference"/>
</flow>
<flow name="MyElementsFlow" processingStrategy="synchronous">
<db:select config-ref="DB-Connector" doc:name="Select">
<db:template-query-ref name="select-template" />
</db:select>
<component class="myJavaComponent" doc:name="To XML"/>
</flow>
edit: I thought I'd try setting the HTTP connector to responseStreamingMode="ALWAYS" and the DB select to streaming="true", but when I do that, the iterator I get in my payload returns false from hasNext(), and next() throws an exception about being closed.
edit: Corrected result of calling getOutputStream.

Answer 1

The problem is that you're trying to load an XML which is unreasonably big into Memory (by using dom), the 10 minutes you're experiencing is probably spent most in garbage collection, so the first trivial thing to do to improve this is to add more heap to your wrapper.conf, you can do this only if the process will run in very low load.
I would suggest you to try a divide and conquer technique, maybe using your disk as intermediate storage.
You can use stax to stream XML
https://docs.oracle.com/javase/tutorial/jaxp/stax/using.html