I have three different systems. I am using Spring integration to sync data in all these system.
System 1 calls --- > System 2 via http:inbound gateway
<int-http:inbound-gateway id="gateway"
path="/save" supported-methods="POST, PUT"
request-channel="requestChannel" reply-channel="replyChannel"
request-payload-type="com.model.Request"
mapped-request-headers="source" error-channel="errorChannel" />
System 2 will call service method to persist data which returns response if request is valid else throws exception
<int:service-activator ref="serviceMethod"
method="saveIfValid" input-channel="requestChannel"
output-channel="serviceOutput" />
<int:recipient-list-router id="id1"
input-channel="serviceOutput">
<int:recipient channel="system1" />
<int:recipient channel="system3" />
</int:recipient-list-router>
I need to send service method response to system 1 and system 3 only if operation is successful.
After calling service method, based on service method response, request for system 3 will be generated using transformer. After transformer I am putting request in mq queue.
<int:transformer id="transformer1"
method="convert" input-channel="system3"
output-channel="jmsInput">
<bean
class="com.transformer.System3Transformer" />
</int:transformer>
<int-jms:outbound-channel-adapter id="adapter"
channel="jmsInput" destination-name="queueName">
</int-jms:outbound-channel-adapter>
Updated JMS outbound code
<int-jms:outbound-channel-adapter id="jms1"
channel="jmsIn" destination-name="queueName">
<int-jms:request-handler-advice-chain>
<bean
class="org.springframework.integration.handler.advice.ExpressionEvaluatingRequestHandlerAdvice">
<property name="onSuccessExpression" value="T(Boolean).TRUE" />
<property name="successChannelName" value="afterSuccessDeleteChannel" />
<property name="onFailureExpression" value="T(Boolean).FALSE" />
<property name="failureChannelName" value="afterFailRenameChannel" />
</bean>
</int-jms:request-handler-advice-chain>
</int-jms:outbound-channel-adapter>
My question is
if service class fails I need to send error response and stop the flow
if service method successful persisted data, but transformation fails, system 1 should get success response and error should be logged.
Here as I am using error channel in outbound adapter, even if error occurs in transformer it is returned to system 1.
please suggest how can i handle error without using global error channel and how to handle error in jms outbound adapter.
thank you for answering my quesion
For the first case you should really just rely on the exception propagation - the flow stops and error is sent to the system1 as an HTTP response.
For the second (transformer) case you should take a look into the ExpressionEvaluatingRequestHandlerAdvice and use it with the <request-handler-advice-chain> for the <transformer>.
The same can be applied for the <int-jms:outbound-channel-adapter>, if you should acknowledge to the system1.
http://docs.spring.io/spring-integration/reference/html/messaging-endpoints-chapter.html#expression-advice
https://github.com/spring-projects/spring-integration-samples/tree/master/intermediate/retry-and-more
Related
We are using the Camel BigQuery API (version 2.20) to stream records from a message queue on an ActiveMQ server (version 5.14.3) into a Google BigQuery table.
We have implemented and deployed the streaming mechanism as an XML route definition in a Spring Framework thus:
<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.springframework.org/schema/beans"
xmlns:beans="http://www.springframework.org/schema/beans"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
./spring-beans.xsd
http://camel.apache.org/schema/spring
./camel-spring.xsd">
<!--
# ==========================================================================
# ActiveMQ JMS Bean Definition
# ==========================================================================
-->
<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="nio://192.168.10.10:61616?jms.useAsyncSend=true" />
<property name="userName" value="MyAmqUserName" />
<property name="password" value="MyAmqPassword" />
</bean>
</property>
</bean>
<!--
# ==========================================================================
# GoogleBigQueryComponent
# https://github.com/apache/camel/tree/master/components/camel-google-bigquery
# ==========================================================================
-->
<bean id="gcp" class="org.apache.camel.component.google.bigquery.GoogleBigQueryComponent">
<property name="connectionFactory">
<bean class="org.apache.camel.component.google.bigquery.GoogleBigQueryConnectionFactory">
<property name="credentialsFileLocation" value="MyDir/MyGcpKeyFile.json" />
</bean>
</property>
</bean>
<!--
# ==========================================================================
# Main Context Bean Definition
# ==========================================================================
-->
<camelContext id="camelContext" xmlns="http://camel.apache.org/schema/spring" >
<!--
========================================================================
https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/processor/RedeliveryPolicy.html
========================================================================
-->
<onException useOriginalMessage="true">
<exception>com.google.api.client.googleapis.json.GoogleJsonResponseException</exception>
<exception>java.net.SocketTimeoutException</exception>
<exception>java.net.ConnectException</exception>
<redeliveryPolicy
backOffMultiplier="2"
logHandled="false"
logRetryAttempted="true"
maximumRedeliveries="10"
maximumRedeliveryDelay="60000"
redeliveryDelay="1000"
retriesExhaustedLogLevel ="ERROR"
retryAttemptedLogLevel="WARN"
/>
</onException>
<!--
# ==================================================================
# Message Route :
# 1. consume messages from my AMQ queue
# 2. write message to Google BigQuery table
# see https://github.com/apache/camel/blob/master/components/camel-google-bigquery/src/main/docs/google-bigquery-component.adoc
# ==================================================================
-->
<route>
<from uri="jms:my.amq.queue.of.output.data.for.gcp?acknowledgementModeName=DUPS_OK_ACKNOWLEDGE&concurrentConsumers=20" />
<to uri="gcp:my_gcp_project:my_bq_data_set:my_bq_table" />
</route>
</camelContext>
</beans>
The above seems to work and we seem to be landing a high of rate messages/records (one route handles over 12,000 message per minute) but our logs are showing a good number of the SocketTimeoutException, 502 Bad Gateway and 500 Internal Server Error warnings :
2019-10-21 15:33:13 | WARN | DefaultErrorHandler | Failed delivery for (MessageId: XXX on ExchangeId: XXX). On delivery attempt: 0 caught: java.net.SocketTimeoutException: connect timed out
2019-10-24 12:46:53 | WARN | DefaultErrorHandler | Failed delivery for (MessageId: XXX on ExchangeId: XXX). On delivery attempt: 0 caught: com.google.api.client.googleapis.json.GoogleJsonResponseException: 502 Bad Gateway
2019-10-25 12:33:33 | WARN | DefaultErrorHandler | Failed delivery for (MessageId: XXX on ExchangeId: XXX). On delivery attempt: 0 caught: com.google.api.client.googleapis.json.GoogleJsonResponseException: 500 Internal Server Error
Questions
Is my use of the onException object generally/syntactically correct (barring the fine-tuning of the redeliveryPolicy attributes) ? Or have I missed out anything else ?
My first warning message of interest says, "On delivery attempt: 0 caught: java.net.SocketTimeoutException". My log file does not have "On delivery attempt: 1", On delivery attempt: 2", etc. Does this mean that subsequent delivery attempts of the given message were successful ?
As far as trying to stream data into GCP is concerned, should I treat the "SocketTimeoutException" "500 Internal Server Error" and "502 Bad Gateway" differently from each other or is using the same onException + redeliveryPolicy OK ?
Are there any other ways I can improve the performance of this Camel / Google API method of streaming data into GCP ? Can the Camel / Google API support message batching in order to reduce the number of GCP insert operations ? I'm already using dual streams with deduplication (CamelGoogleBigQueryInsertId).
Disclaimer: I don't have experience in using Camel BigQuery API. My answer is based on observation and understanding of BigQuery API in general.
Based on observation that there is retriesExhaustedLogLevel ="ERROR", if no ERROR log presents, it probably means retry succeeded.
Retry on timeout/500/502 can be the same. At least I'm not aware of how they can be treated differently.
Batching will definitely help, based on public documentation:
Maximum rows per request: 10,000 rows per request
A maximum of 500 rows is recommended. Batching can increase
performance and throughput to a point, but at the cost of per-request
latency. Too few rows per request and the overhead of each request can
make ingestion inefficient. Too many rows per request and the
throughput may drop.
A maximum of 500 rows per request is recommended, but experimentation with representative data (schema and data sizes) will help you determine the ideal batch size.
I've faced an issue, with trying to forward message, returned from my custom REST end-point, into JMS queue.
I wanted to implement something like:
SMTH CUSTOM IS PUSHING MESSAGE TO WSO2 JMS QUEUE called testQueue
WSO2 is retrieving this message, and once there is an message, it's pushing this message next to custom REST service:
Becasue I've defined receive to testResponse inside send section, I'm able to log reponse from REST API inside testResponse.
Now I'm stuck, as I'm trying to somehow push response from REST API next, to let say second JMS queue, called testQueue2.
I've already tried with call tags inside my custom sequence (called testReponse), but it's just not executing at all (I've also tried for test, just to execute second REST API inside, to be just sure, if it's not something with executing of JMS itself...):
Any ideas, how I can move response from called REST API on to JMS queue?
Thanks.
<proxy name="testQueue" startOnLoad="true" transports="jms">
<target>
<inSequence>
<send receive="testResponse">
<endpoint>
<http format="rest" method="POST" uri-template="http://localhost/customRestAPI?message=test" />
</endpoint>
</send>
</inSequence>
<outSequence />
</target>
</proxy>
<sequence name="testResponse">
<log level="full" />
</sequence>
Try adding an "Out Flow" mediation extension as explained here.
I wish all my Proxy Services to log a standard ERROR when the endpoint publishes a soapfault. Setting faultSequence to a valid logging sequence doesn't seem to be doing anything. I know the endpoint is publishing fault responses & the faultlogger sequence is a simply FULL log.
<SOAP-ENV:Fault...> </SOAP-ENV:Fault>
<proxy ... >
<target faultSequence="faultlogger" endpoint="conf:/myService>
...
</target>
</proxy>
How does faultSequence work?
I also tried to use a out, filter + log but wasn't sure what the xpath expression I should use to filter for faults.
faultSequence is executed when there is a technical error (in your xpath for exemple, or can't connect to an address configured in an endpoint)
A SOAPFault is not a technical problem and the out sequence is executed : you can verify response message to find potentials soap faults.
An other solution is to set the property named FORCE_ERROR_ON_SOAP_FAULT :
<property name="FORCE_ERROR_ON_SOAP_FAULT" value="true"/>
In the spring integration reference on JDBC Support
it has the following code to get the records out of a table and convert them into message and sent them to a channel. But my system is rabbitmq so I want the message from jdbc stored in rabbitmq and consumed by rabbitmq client. How to do that using spring integration and spring rabbitmq?
<int-jdbc:inbound-channel-adapter query="select * from item where status=2"
channel="target" data-source="dataSource"
update="update item set status=10 where id in (:id)" />
I know there are AMQP Backed Message Channels, and defined as below.
<int-amqp:channel id="p2pChannel"/>
But how to put them together? So that the message created from a table will be stored in a rabbitmq queue?
update: I used the following code:
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="org.postgresql.Driver"/>
<property name="url" value="jdbc:postgresql://localhost/example"/>
<property name="username" value="postgres"/>
<property name="password" value="oracle"/>
</bean>
<int-amqp:channel id="myMessageChannel_test"/>
<int-jdbc:inbound-channel-adapter query="select * from my_files where message_created=0"
channel="myMessageChannel_test" data-source="dataSource"
update="update my_files set message_created=1 where file_id in (:file_id)">
<int:poller fixed-rate="1000">
<int:transactional/>
</int:poller>
</int-jdbc:inbound-channel-adapter>
But it complains:
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.messaging.MessageDeliveryException: Dispatcher has no subscribers for amqp-channel 'myMessageChannel_test'.; nested exception is org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:197)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:737)
... 10 common frames omitted
Caused by: org.springframework.integration.MessageDispatchingException: Dispatcher has no subscribers
at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:107)
at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:97)
at org.springframework.integration.amqp.channel.AbstractSubscribableAmqpChannel$DispatchingMessageListener.onMessage(AbstractSubscribableAmqpChannel.java:181)
... 11 common frames omitted
2014-12-30 19:23:18.985 WARN 29203 --- [cTaskExecutor-1] s.a.r.l.ConditionalRejectingErrorHandler : Execution of Rabbit message listener failed.
org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException: Listener threw exception
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:802)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:740)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:628)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:167)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1196)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:600)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:960)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:944)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$700(SimpleMessageListenerContainer.java:82)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1058)
So your channel "myMessageChannel_test" has messages from database but that needs some config like below to be able to receive/handle/subscribe to messages.
There are many ways to do it - one way as below
<rabbit:connection-factory id="rabbitConnFactory" addresses="${rabbit.addresses}"
virtual-host="${rabbit.host}" username="${rabbit.username}" password="${rabbit.password}"/>
<rabbit:admin id="admin" connection-factory="rabbitConnFactory" />
<rabbit:template id="amqpTemplate" connection-factory="rabbitConnFactory" routing-key="some.key" exchange="some.exchange" queue="some.queue" />
<int-amqp:outbound-channel-adapter routing-key="some.key" amqp-template="amqpTemplate"
channel="myMessageChannel_test" />
You need to do respective rabbit configs as well
I'm newbie in Spring Integration Framework but I try to understand what is wrong in my configuration since several days, and I'm blocked ...
Perhaps you can see what is wrong.
It's pooling files from a directory and transforming the File Payload into a LaunchRequest (to launch a Spring Bath job).
My program detects when a file is created in the directory but doesn't send the file in the 'files' channel (MessageDeliveryException : failed to send message to channel 'files')
[...]
<int:service-activator method="launch" input-channel="files-requests"
output-channel="statuses">
<bean class="org.springframework.batch.integration.launch.JobLaunchingMessageHandler">
<constructor-arg ref="jobLauncher"/>
</bean>
</int:service-activator>
<int:channel id="files"/>
<int:channel id="files-requests"/>
<int:channel id="statuses"/>
[...]
Thank you for your help