Dynamic SELECT query which decides whether to use the WHERE clause in Mule 4 - mule

I am using Mule 4 and Anypoint 7 and want to setup the database connector to SELECT all customers from my SQL server database table but if the customerName query parameter is populated in the request then I want to add the WHERE clause to only return customers with the same name as the customerName query parameter otherwise it should just return all customers.
My code is below but I am struggling to get the syntax correct.
<db:select doc:name="Select Customers" doc:id="98a4aa2f-b0b6-4fb5-ab27-d70489fd532d" config-ref="db-config">
<ee:repeatable-file-store-iterable />
<db:sql >SELECT TOP 10 * FROM MYDB.dbo.Customer $(if (attributes.queryParams.customerName != null and isEmpty(attributes.queryParams.customerName) == false) "WHERE Name = :customerName" else "")</db:sql>
<db:input-parameters ><![CDATA[#[{'customerName' : attributes.queryParams.customerName}]]]></db:input-parameters>
</db:select>
How can I do this?
Thanks

You were on the right path. I think you were only missing the evaluation tags around the SQL in the db:sql element.
<db:select doc:name="Select Customers" doc:id="98a4aa2f-b0b6-4fb5-ab27-d70489fd532d"
config-ref="db-config">
<ee:repeatable-file-store-iterable />
<db:sql>#["SELECT TOP 10 * FROM MYDB.dbo.Customer
$(if (isEmpty(attributes.queryParams.customerName) == false) "WHERE Name = :customerName" else "") "]</db:sql>
<db:input-parameters ><![CDATA[#[{'customerName' : attributes.queryParams.customerName}]]]></db:input-parameters>
</db:select>
It is easier to debug things like this with variables, so that you can see the individual values. FWIW heres my test code:
<set-variable variableName="additionalWhereClause"
value='#[if ( isEmpty(attributes.queryParams.email) == false)
"WHERE Email = :emailParm"
else "" ]' />
<set-variable variableName="selectSql"
value="#['SELECT FirstName, LastName, Email
FROM User
$( vars.additionalWhereClause )
ORDER BY Email LIMIT 10']" />
<logger level="INFO" message="queryParams: #[attributes.queryParams]" doc:id="96c62f84-2c98-4df6-829c-e00c9fcec9ca" />
<logger level="INFO" message="additionalWhereClause #[vars.additionalWhereClause]" doc:id="0d3611b4-34ae-4ebb-b931-6d31ce3804c1" />
<logger level="INFO" message="selectSql #[vars.selectSql]" doc:id="5c56342d-9674-4891-9d7e-bb32319f4ad0" />
<db:select doc:name="MySQL Query" doc:id="e60be3e6-9b51-4b3b-9dfa-4ee0af65cb03"
config-ref="mysql-config">
<ee:repeatable-file-store-iterable />
<db:sql>#[ vars.selectSql ]</db:sql>
<db:input-parameters><![CDATA[#[{'emailParm' : attributes.queryParams.email}]]]></db:input-parameters>
</db:select>

Related

how to avoid long sql in clickhouse

As ClickHouse can run complex SQL efficiently, we now write hundred-lines-in-one SQL to get the real time analysis results(join lots of source table). And it brings a problem is that it's so hard to maintain and reuse.
And we are using mybatis to write the sql in our Java Application. We try to use Mybatis's global parameter to define some sql for reuse, but not helps a lot in such huge lines SQL.
Is there any way to avoid such long sql or make the sql more readable and make this work more engineerable?
<select id="brandAnalysis" resultType="java.util.HashMap">
SELECT gmvRank.id_code,
gmvRank.rank,
gmvRank.lastRank,
gmvRank.gmv as sale_money,
member.dealerMembers as dealer_turnover,
member.teamMembers,
case
when member.dealerMembers = 0 then 0
when member.teamMembers = 0 then 0
else member.dealerMembers/member.teamMembers end AS dealer_turnover_active
FROM (
SELECT id_code,
<choose>
<when test="monthQuarter == 3">
case
when max(`quarter`) = #{quarter} then argMax(${sqlRankField},`month`)
else null end as rank,
case
when max(`quarter`) = #{quarter} then argMax(${sqlGmvField},`month`)
else null end as gmv,
case
when min(`quarter`) = #{compareQuarter} then argMin(${sqlRankField},`month`)
else null end as lastRank
</when>
<otherwise>
case
when max(`month`) = #{month} then argMax(${sqlRankField},`month`)
else null end as rank,
case
when max(`month`) = #{month} then argMax(${sqlGmvField},`month`)
else null end as gmv,
case
when min(`month`) = #{compareMonth} then argMin(${sqlRankField},`month`)
else null end as lastRank
</otherwise>
</choose>
FROM
tengju.dws_crm_brand_analysis_rank_dd_m
<choose>
<when test="monthQuarter == 3">
PREWHERE quarter in (#{quarter},#{compareQuarter})
and insert_time in (
SELECT MAX(insert_time)
FROM tengju.dws_crm_brand_analysis_rank_dd_m
PREWHERE quarter in (#{quarter},#{compareQuarter})
GROUP BY quarter
)
</when>
<otherwise>
PREWHERE month in (#{month},#{compareMonth})
and insert_time in (
SELECT MAX(insert_time)
FROM tengju.dws_crm_brand_analysis_rank_dd_m
PREWHERE month in (#{month},#{compareMonth})
GROUP BY month
)
</otherwise>
</choose>
<if test="dominationIdCode != null and dominationLevel != null">
and id_code in (
SELECT id_code
FROM tengju.dwd_user_domination_relation_map_all
PREWHERE visit_date >= formatDateTime(yesterday(),'%F')
AND insert_time in (
SELECT max(insert_time)
FROM tengju.dwd_user_domination_relation_map_all
PREWHERE visit_date >= formatDateTime(yesterday(),'%F')
)
AND domination_star_level = #{dominationLevel}
AND domination_id_code = #{dominationIdCode}
<if test="starLevels != null and starLevels.size() > 0">
AND star_level in
<foreach collection="starLevels" item="item" index="index" open="(" close=")" separator=",">
#{item}
</foreach>
</if>
<if test="isDirect != null and isDirect > 0">
AND parent_id_code = #{dominationIdCode}
</if>
)
</if>
<if test="idCodes != null and idCodes.size() > 0">
AND id_code in
<foreach collection="idCodes" item="item" index="index" open="(" close=")" separator=",">
#{item}
</foreach>
</if>
AND brand_id = #{brandId}
GROUP BY id_code
) as gmvRank
GLOBAL LEFT JOIN (
<choose>
<when test="crmRankCrowdPerspective == 4">
<include refid="smallTeamDealer" />
</when>
<when test="crmRankCrowdPerspective == 1">
<include refid="KATeamDealer" />
</when>
<otherwise>
<include refid="ecOrCityTeamDealer" />
</otherwise>
</choose>
) as member
on (gmvRank.id_code = member.id_code)
where 1=1
<if test="saleMoneyFrom != null">
AND gmvRank.gmv >= toDecimal128(#{saleMoneyFrom},2)
</if>
<if test="saleMoneyTo != null">
AND toDecimal128(#{saleMoneyTo},2) >= gmvRank.gmv
</if>
order by ${sortType}
LIMIT #{offset},#{pageSize}
</select>
User common table expressions to get rid of nested operations. Each make and version of database server has its own style of SQL. Not all have common table expressions. All have views though. SQL queries can be insanely verbose, for sure there is no other option.

How to stop executing flow reference unless and until the complete batch is finished?

In my project I am reading a huge tsv file (around 820k) and process the records into db. I have a java file which does validation, after the validation I store the records in a list. Since I can't send the whole list to the vm,I split the list into 2000 and send to the flow. here is how I send 2000 records
while(i < validRecords.size()){
int j = i + 2000;
if( j < validRecords.size())
{
muleClient.dispatch("vm://validRecordsEtoc", validRecords.subList(i, j), msgProperties);
}
else
{
muleClient.dispatch("vm://endRecordsEtoc", validRecords.subList(i,validRecords.size()-1), msgProperties);
}
i = i + 2000;
}
I have a flow which has reference to a batch and process the records into db.
Here is the flow
<flow name="csv-source-inputFlow_Clean_etoc">
<vm:inbound-endpoint exchange-pattern="one-way" path="validRecordsEtoc" doc:name="VM_TFO_ETOC_SUBS_SRC"/>
<enricher doc:name="Enrich inbound Message Properties">
<logger message="Etoc source message enricher : BatchId : #[message.inboundProperties.flow_batch_id] Total Records : #[message.inboundProperties.totalRecordCount] Successful Records : #[message.inboundProperties.successRecordCount]" level="INFO" doc:name="Logger"/>
<enrich source="#[message.inboundProperties.'flow_batch_id']" target="#[flowVars.flow_batch_id]"/>
<enrich source="#[message.inboundProperties.'flow_source_name']" target="#[flowVars.flow_source_name]"/>
<enrich source="#[message.inboundProperties.'successRecordCount']" target="#[flowVars.successRecordCount]"/>
<enrich source="#[message.inboundProperties.'totalRecordCount']" target="#[flowVars.totalRecords]"/>
<enrich source="#[message.inboundProperties.'input_file_name']" target="#[flowVars.input_file_name]"/>
</enricher>
<batch:execute name="csv-source-inputBatch_ETOC" doc:name="csv-source-inputBatch_ETOC"/>
</flow>
<batch:job name="csv-source-inputBatch_ETOC">
<batch:input>
<set-payload value="#[payload]" doc:name="Set Payload_TFO_ETOC"/>
</batch:input>
<batch:process-records>
<batch:step name="TFOETOC_CLEAN_Batch_Step">
<batch:commit size="2000" doc:name="TFO_ETOC Clean Batch Commit">
<db:insert config-ref="LocalhostPostgres" bulkMode="true" doc:name="Enrich_TFO_ETOC_Src_Data">
<db:parameterized-query><![CDATA[INSERT INTO public."csv_tfo_3.4.1 etoc subs"(
seq_id, batch_id, publication, code, first_name,
last_name,subscription_date, email, marketable ,
organization, department, address1 ,
address2, city, state,
zip_code, country, phone,
status, source)
values (DEFAULT,#[flowVars.flow_batch_id],#[payload.Publication] , #[payload.Code] , #[payload.First_Name],#[payload.Last_Name], to_timestamp(#[payload.Subscription_Date],'yyyy-mm-dd hh24:mi:ss'),#[payload.Email],#[payload.Marketable],#[payload.Organization],#[payload.Department],#[payload.Address1],#[payload.Address2],#[payload.City],#[payload.State],#[payload.Zip_Code],#[payload.Country],#[payload.Phone],'pending','TFO')]]></db:parameterized-query>
</db:insert>
</batch:commit>
</batch:step>
</batch:process-records>
</batch:job>
when I process the last list , i am sending to another flow because i need to call the sub-flow in on complete, where as my last vm gets executed before the previous lists get executed.
I followed the below link
How to read huge CSV file in Mule
Can anyone give me any idea how to do that?.

How to write dependent queries in MULE CE?

I hear that we con't write two Inbound-endpoints in one flow, but below is my requirement, Second select query needs first query payload value. When I run this giving an exception. Please let me know if there is any alternate way to write this.
<flow name="Some">
<jdbc:inbound-endpoint queryKey="SelectSome"
connector-ref="ProConnector" doc:name="SomeFromPro"
pollingFrequency="10000" queryTimeout="-1">
<jdbc:transaction action="NONE" timeout="10" />
<jdbc:query key="SelectSome"
value="SELECT top 1 * from table1 where IsProcessed = 0" />
<jdbc:query key="SelectSome.ack"
value="update table1 set IsProcessed=1 where ID = #[map-payload:ID] " />
</jdbc:inbound-endpoint>
<jdbc:inbound-endpoint queryKey="SelectSomeBR"
connector-ref="ProConnector" doc:name="SomeBRFromPro"
pollingFrequency="1000" queryTimeout="-1">
<jdbc:transaction action="NONE" timeout="10" />
<jdbc:query key="SelectSomeBR"
value="SELECT * from table2 where IsProcessed = 0 and ParentID = #[map-payload:ID]" />
<jdbc:query key="SelectSomeBR.ack"
value="update table2 set IsProcessed=1 where ParentID = #[map-payload:ID] " />
</jdbc:inbound-endpoint>
.
.
.
.
</flow>
Any help would be appreciated.
Thank you.
You need to use an outbound select query for your second query, like:
<jdbc:outbound-endpoint queryKey="SelectSomeBR"
connector-ref="ProConnector" doc:name="SomeBRFromPro"
pollingFrequency="1000" queryTimeout="-1"
exchange-pattern="request-response">
<jdbc:transaction action="NONE" timeout="10" />
<jdbc:query key="SelectSomeBR"
value="SELECT * from table2 where IsProcessed = 0 and ParentID = #[map-payload:ID]" />
<jdbc:query key="SelectSomeBR.ack"
value="update table2 set IsProcessed=1 where ParentID = #[map-payload:ID] " />
</jdbc:outbound-endpoint>
Reference: https://developer.mulesoft.com/docs/display/current/JDBC+Transport+Reference#JDBCTransportReference-OutboundSELECTQueries
PS. This old expression syntax is obsolete: #[map-payload:ID] Use MEL instead: #[message.payload.ID]" (assuming the message payload is a map with a key named ID).

replace xml particular node element value with other value in mule

<healthcare>
<plans>
<plan1>
<planid>100</planid>
<planname>medical</planname>
<desc>medical</desc>
<offerprice>500</offerprice>
<area>texas</area>
</plan1>
<plan2>
<planid>101</planid>
<planname>dental</planname>
<desc>dental</desc>
<offerprice>1000</offerprice>
<area>texas</area>
</plan2>
</plans>
</healthcare>
<splitter evaluator="xpath" expression="/healthcare/plans" doc:name="Splitter"/>
<transformer ref="domToXml" doc:name="Transformer Reference"/>
<logger level="INFO" doc:name="Logger" message=" plans detils...#[message.payload]" />
i want replace offerprice value with other values during runtime.anyhelp appreciated.I tried various various ways . anyone shed light means it saves me lot
You could use XSLT and use identity templates to replace the one element.Or if your really want to do it with MEL, convert to DOM and use Dom4j APIs to set the value and then convert back to XML if needed:
<expression-component><![CDATA[
node = message.payload.getRootElement().selectSingleNode('//plans/plan1/planid');
node.text = 'newvalue';
]]></expression-component>
<mulexml:dom-to-xml-transformer />
<logger level="ERROR" message=" #[payload]" />
EDIT
Here is an example if you want to update multiple nodes. If the transformation gets any more complex, I would really suggest taking a look at XSLT.
<mulexml:xml-to-dom-transformer returnClass="org.dom4j.Document" />
<expression-component><![CDATA[
plans = message.payload.getRootElement().selectNodes('//plans/*');
foreach (plan : plans){
plan.selectSingleNode('offerprice').text = '3000';
} ]]></expression-component>
<mulexml:dom-to-xml-transformer />
<logger level="ERROR" message=" #[payload]" />

Passing date parameter to jdbc query in Mule

I have a flow in Mule where I want to use a date parameter i get from one query as an input for another query.
<jdbc:connector name="myConnector" transactionPerMessage="false" dataSource-ref="myDataSource">
<jdbc:query key="getPollTimes" value="SELECT to_char(last_poll_start, 'YYYY-MM-DD HH24:MI:SS') as last_poll_start, to_char(last_poll_end, 'YYYY-MM-DD HH24:MI:SS') as last_poll_end FROM db_sources WHERE source_system = 'mySystem'" />
<jdbc:query key="getCustomerIds" value="SELECT id FROM customers WHERE updated < TO_DATE(#[variable:last_poll_end],'YYYY-MM-DD HH24:MI:SS')" />
</jdbc:connector>
<flow name="myFlow">
<enricher>
<jdbc:outbound-endpoint queryKey="getPollTimes" exchange-pattern="request-response" />
<enrich target="#[variable:last_poll_end]" source="#[groovy:payload.last_poll_end]"/>
</enricher>
<logger level="INFO" message="last_poll_end = #[variable:last_poll_end]" />
<jdbc:outbound-endpoint queryKey="getCustomerIds" exchange-pattern="request-response" />
</flow>
When running this I cannot get this to work (note that I am using an Oracle DB). I have included the exception below. Have anyone encountered this?
--------------------------------------------------------------------------------
Exception stack is:
1. Invalid column type(SQL Code: 17004, SQL State: + null) (java.sql.SQLException)
oracle.jdbc.driver.DatabaseError:113 (null)
2. Invalid column type Query: SELECT ID FROM CUSTOMERS WHERE UPDATED < TO_DATE(?,'YYYY-MM-DD HH24:MI:SS') Parameters: [[2000-01-01]](SQL Code: 17004, SQL State: + null) (java.sql.SQLException)
org.apache.commons.dbutils.QueryRunner:540 (null)
3. Failed to route event via endpoint: DefaultOutboundEndpoint{endpointUri=jdbc://getCustomerIds, connector=JdbcConnector
{
name=myConnector
lifecycle=start
this=668e94
numberOfConcurrentTransactedReceivers=4
createMultipleTransactedReceivers=false
connected=true
supportedProtocols=[jdbc]
serviceOverrides=<none>
}
, name='endpoint.jdbc.getCustomerIds', mep=REQUEST_RESPONSE, properties={queryTimeout=-1}, transactionConfig=Transaction{factory=null, action=INDIFFERENT, timeout=0}, deleteUnacceptedMessages=false, initialState=started, responseTimeout=10000, endpointEncoding=UTF-8, disableTransportTransformer=false}. Message payload is of type: ArrayList (org.mule.api.transport.DispatchException)
org.mule.transport.AbstractMessageDispatcher:106 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/transport/DispatchException.html)
--------------------------------------------------------------------------------
Problem is solved. The issue was partly that the date variable I got back from the first query was stored as an array. To resolve this i pick out the first element. Besides that I removed the to_date() in the second sql query.
This gets the first element in the array:
<enrich target="#[variable:last_poll_end]" source="#[groovy:payload.last_poll_end[0]]"/>
The updated sql:
<jdbc:query key="getCustomerIds" value="SELECT id FROM customers WHERE updated < #[variable:last_poll_end]" />