Mulesoft: Insert into sql server fails b/c of null constraint however payload has data - sql

Question

I'm still new to Mulesoft's Mule ESB and I just cannot get past this issue.
I'm having trouble understanding why my Mule flow is failing with a sql server insert error.
It appears that the payload has data but for some reason the query parameters are not being populated with that data. I can read from a table or file fine it's just the writing to a db table that goes sideways.
What do I need to do to write my payload to a sql server database?
Below is the error message I get when I run my Mule Application in AnyPoint Studio. I've removed the bulk of the payload from the message due to privacy.
I've also posted my flow xml.
Any help would be appreciated. Thanks
********************************************************************************
Message : Cannot insert the value NULL into column 'AgentId', table 'GARDB1dev.src.AgentList'; column does not allow nulls. INSERT fails. (com.microsoft.sqlserver.jdbc.SQLServerException).
Payload : [{Agent_ID=10032, **REDACTED DATA** [..]]
Payload Type : java.util.ArrayList
Element : /Load-AgentList/processors/2/1/1 # gar-data-load:sharepoint.xml:51
Element XML : <db:insert config-ref="GAR-DB-Connection-SSPI" doc:name="Insert AgentList Data">
<db:parameterized-query>insert into src.AgentList (AgentId,PartyId,PartyType,AgentIdStatus,SalesOrg,AgentIdStatusStart,AgentIdStatusEnd,AgentName,SubOrganization,[Function],NewToOrg,NewToChannel,DaysAgentIdActive,Region,AgentPhone,CertificationSummary)Values (#[payload.Agent_ID],#[payload.Party_ID],#[payload.Party_Type],#[payload.Agent_ID_Status],#[payload.Sales_Organization],#[payload.Agent_ID_Status_Start],#[payload.Agent_ID_Status_End],#[payload.Agent_Name],#[payload.Sub_Organization],#[payload.Function],#[payload.New_To_Org],#[payload.New_To_Channel],#[payload.Days_Agent_ID_Active],#[payload.Region],#[payload.Agent_Phone],#[payload.Certification_Summary])</db:parameterized-query>
</db:insert>
--------------------------------------------------------------------------------
The XML description of my Mule Flow is the following:
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:db="http://www.mulesoft.org/schema/mule/db" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:metadata="http://www.mulesoft.org/schema/mule/metadata" xmlns:sharepoint2010="http://www.mulesoft.org/schema/mule/sharepoint2010" xmlns:sharepoint="http://www.mulesoft.org/schema/mule/sharepoint" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/sharepoint http://www.mulesoft.org/schema/mule/sharepoint/current/mule-sharepoint.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/sharepoint2010 http://www.mulesoft.org/schema/mule/sharepoint2010/current/mule-sharepoint2010.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd
http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
<file:connector name="input" autoDelete="false" streaming="true" validateConnections="true" doc:name="File" readFromDirectory="C:\Users\rpearso7\_gar-data\input" />
<flow name="Load-AgentList">
<file:inbound-endpoint path="C:\Users\rpearso7\_gar-data\input" responseTimeout="10000" doc:name="File" moveToDirectory="C:\Users\rpearso7\_gar-data\archive" connector-ref="input" moveToPattern="#[function:datestamp]-#[message.inboundProperties['originalFilename']]">
<file:filename-regex-filter pattern="agent_list\.csv" caseSensitive="false"/>
</file:inbound-endpoint>
<dw:transform-message doc:name="Transform Message" metadata:id="fdb1aea0-6e94-425f-9493-ee71983b8eb1">
<dw:input-payload mimeType="application/csv"/>
<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
payload]]></dw:set-payload>
</dw:transform-message>
<logger message="************** Payload Loaded" level="INFO" doc:name="Log Payload"/>
<scatter-gather doc:name="Scatter-Gather">
<threading-profile maxThreadsActive="1" poolExhaustedAction="WAIT"/>
<processor-chain>
<logger message="***************************** Truncate AgentList Table" level="INFO" doc:name="Logger"/>
<db:update config-ref="GAR-DB-Connection-SSPI" doc:name="Truncate table">
<db:parameterized-query><![CDATA[truncate table src.AgentList;]]></db:parameterized-query>
</db:update>
<logger message="************************************ AgentList table truncated" level="INFO" doc:name="Logger"/>
</processor-chain>
<processor-chain>
<logger message="********************************* Start Insert" level="INFO" doc:name="Logger"/>
<db:insert config-ref="GAR-DB-Connection-SSPI" doc:name="Insert AgentList Data">
<db:parameterized-query><![CDATA[insert into src.AgentList (
AgentId,
PartyId,
PartyType,
AgentIdStatus,
SalesOrg,
AgentIdStatusStart,
AgentIdStatusEnd,
AgentName,
SubOrganization,
[Function],
NewToOrg,
NewToChannel,
DaysAgentIdActive,
Region,
AgentPhone,
CertificationSummary)
Values (
#[payload.Agent_ID],
#[payload.Party_ID],
#[payload.Party_Type],
#[payload.Agent_ID_Status],
#[payload.Sales_Organization],
#[payload.Agent_ID_Status_Start],
#[payload.Agent_ID_Status_End],
#[payload.Agent_Name],
#[payload.Sub_Organization],
#[payload.Function],
#[payload.New_To_Org],
#[payload.New_To_Channel],
#[payload.Days_Agent_ID_Active],
#[payload.Region],
#[payload.Agent_Phone],
#[payload.Certification_Summary])]]></db:parameterized-query>
</db:insert>
</processor-chain>
</scatter-gather>
<logger message="**************** Complete" level="INFO" doc:name="Logger"/>
</flow>
</mule>

Answer 1

I have had a look what happens in the debugger, when I use your transformation on a CSV file.
I have set a breakpoint at the first logger, so I could stop the execution and have a look how different Mule Expression Language (MEL) expressions behave in different steps of your flow.
When I evaluate #[payload] after the transformation; I get an ArrayList with the lines of the CSV file.
This seems to be correct.
However... Since the scather-gather only sends a copy to each the same payload is visible in the debugger after the scather-gather as well.
For details see the following blog post: Scatter-Gather in Mule ESB
Scatter-Gather is a routing message processor in Mule ESB runtime that
sends a request message to multiple targets concurrently. It then
collects the responses from all routes and aggregates them back into a
single response.
According to this the DB INSERT processing-step in your flow gets an ArrayList. The elements of the ArrayList can be indexed with a null-based integer index. You can't use the names of the columns of the CSV file to use them.
First you have to select an entry from the ArrayList and then you can use the column headers from the CSV file on the entries of the ArrayList, like in the following example:
#[payload[0].Agent_id]
This MEL expression returns the Agent ID as expected. But if you try to execute #[payload.Agent_id] ; you get null.
In my opinion you do exactly this in your INSERT. Instead of inserting each of the entries of the ArrayList created from the CSV file, you try to access the columns of the CSV file on the ArrayList and not on its elements.
A potential solution could be to add a foreach scope.
Using this you could iterate through the entries of the ArrayList and insert each of them to your database table.
Please also see the results of different MEL expressions in the different states in a similfied version of you flow:
(Please note, that the Mule Expression Language is case-sensitive. If the "ID" substring is replaced with "Id", then I get "null" back.)
I have been testing with a simple CSV file with the following contents:
Agent_ID,Agent_Name
7,james bond
1,Mr. 47
8,Dr. X
You could consider a solution similar to the following one:
I have just added a foreach scope.
This flow has generated the following console-output:
INFO 2018-03-14 16:08:37,736 [[testproject01].connector.file.mule.default.receiver.01] org.mule.transport.file.FileMessageReceiver: Lock obtained on file: X:\SomePathToTheFile\agent_list.csv
INFO 2018-03-14 16:08:38,485 [[testproject01].Load-AgentList.stage1.02] org.mule.api.processor.LoggerMessageProcessor: ************** Payload Loaded
INFO 2018-03-14 16:08:38,488 [[testproject01].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: ***************************** Truncate AgentList Table
INFO 2018-03-14 16:08:38,506 [[testproject01].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: ********************************* Start Insert
INFO 2018-03-14 16:08:38,506 [[testproject01].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: ********************************* Start Insert
INFO 2018-03-14 16:08:38,507 [[testproject01].ScatterGatherWorkManager.01] org.mule.api.processor.LoggerMessageProcessor: ********************************* Start Insert
I hope this is what you are trying to achieve.
If you have furhter questions or I misunderstood your goal, then please add a comment.
Thank you very much.

Answer 2

We have experienced deadlock issues in a similar scenarion, when another process is running an INSERT statement against that same database table.
Scenario:
TRUNCATE is currently being executed on a table, when the same time, concurrently another process INSERTS some data in the same table of the database.
First prioritize the TRUNCATE and acquire lock on the table. Once it is done then go for the INSERT.
You should request a table lock before you execute the TRUNCATE.
If you do this you can't get a deadlock -- the table lock won't be granted before the INSERT finishes and once you have the lock another INSERT can't occur.
Update from the comment:
You can use the LOCK TABLE command.