Apache Camel - File Transfer on Particular Route Call - apache

I'm trying to create a webservice which, when called look into a local directory picks up files from there and upload to the ftp server.
I'm able to create a simple route which picks file from local directory and uploads to ftp server below is the code :
<route>
<from uri="file://D:\\FTPTest?noop=true&delay=2000" />
<to uri="ftp://user#host.in:21/public_html/EnterpriseProject?password=password123#"/>
<to uri="bean:myBean?method=test" />
</route>
But, I want to this file transfer to be called when a particular route is called via restlet webservice is called, I tried with the following code, but it didn't work :
<route>
<from uri="direct:fileTransferRoute" />
<to uri="file://D:\\FTPTest?noop=true&delay=2000" />
<to uri="ftp://user#host.in:21/public_html/EnterpriseProject?password=password123#"/>
</route>
The above route is called by restlet from following route :
<route>
<from
uri="restlet:http://0.0.0.0:9080/csitec/{serviceName}?restletMethod=post" />
<process ref="serviceRouteProcessor" />
<toD uri="direct:${in.header.nextRoute}" />
</route>
Here's the code of my serviceRouteProcessor :
public void process(Exchange exchange) throws Exception {
String body = exchange.getIn().getBody(String.class);
String serviceName = exchange.getIn().getHeader(Constants.SERVICE_NAME).toString();
String nextRoute = serviceName+Constants.NEXT_ROUTE_APPENDER;
exchange.getOut().setHeader(Constants.NEXT_ROUTE, nextRoute);
exchange.getOut().setBody(body);
}
Please help me and suggest the changes needs to be done to make it work like this.

You should try the pollEnrich feature of content-enricher
In the example section you can find a example regarding files.
Your route should look something like this(I work only with camel java dsl, so this a bit xml pseudo code):
<route>
<from uri="direct:fileTransferRoute" />
<pollEnrich uri="file://D:\\FTPTest?fileName=data.txt....." />
<to uri="ftp://user#host.in:21/public_html/EnterpriseProject?password=password123#"/>
</route>

Edited :
you must understand one thing first , to is producer not consumer <to uri="file://D:\\FTPTest?noop=true&delay=2000" />
What you can do is ,
#Autowired
private CamelContext context;// if you have more than one camel context use #Qualifier and wire by bean id
public void process(Exchange exchange) throws Exception {
String body = exchange.getIn().getBody(String.class);
String serviceName = exchange.getIn().getHeader(Constants.SERVICE_NAME).toString();
context.startRoute(serviceName+Constants.NEXT_ROUTE_APPENDER);// here in nextroute you must give the routeid
}
your route must look like
<route id = "<value of serviceName+Constants.NEXT_ROUTE_APPENDER>" autoStartup = "false">
<from uri="file://D:\\FTPTest..." />
<onCompletion onFailureOnly="true">
<choice>
<when>
<simple>${property.CamelBatchComplete}</simple>
<process ref="asyncSelfShutdownProcessor"/>
</when>
</choice>
</onCompletion>
<to uri="ftp://user#host.in:21..."/>
</route>
And add asyncSelfShutdownProcessor to spring context
#Component
public class AsyncSelfShutdownProcessor implements AsyncProcessor {
#Autowired
private CamelContext context
public boolean process(Exchange exchange, AsyncCallback callback){
new Thread(() -> context.stopRoute(exchange.getFromRouteId())).start();
}
}
##############################################################################
Old :
OK I understand your need as - you have a route that moves file from file system to ftp server, all you need is this route to get executed only when you trigger from a rest service. I would do it like this ,
*I will make the route autoStartup = "false" and assign as id = "fs-to-ftp" to the route
<route id = "fs-to-ftp" autoStartup = "false">
<from uri="file://D:\\FTPTest..." />
<onCompletion onFailureOnly="true">
<process ref="asyncSelfShutdownProcessor"/>
</onCompletion>
<to uri="ftp://user#host.in:21..."/>
</route>
**Add a self shutdown async process in onComplete to the route "fs-to-ftp". Async Processor
asyncSelfShutdownProcessor= AsyncProcessorConverterHelper.convert(exchange -> {
new Thread(() -> context.stopRoute("fs-to-ftp")).start();
});
***Add the camel context dependency to rest service and start the route by id in the rest service context.startRoute("fs-to-ftp")

Related

Read properties from database and use it instead of mule-app.properties

I have been able to read the properties from a table in the database as it was described here Reading mule config from database
Now, I am not able to apply these properties to the flow configs and also access them as out bound properties in the Java Processor classes through the MuleEventContext's message.
Update: below is my flow XML code
<flow name="push-data">
<poll doc:name="Push Poll">
<fixed-frequency-scheduler frequency="${push.data.poll.frequency}" timeUnit="MINUTES" />
<set-property propertyName="tempFilePath" value="${temp.csv.file.path}" doc:name="Property"/>
<component class="com.reports.processors.PushDataProcessor" doc:name="PushDataProcessor"/>
<logger message="worked!!!" level="INFO" doc:name="Logger"/>
<exception-strategy ref="push-report-data_Catch_Exception_Strategy" doc:name="Reference Exception Strategy"/>
</flow>
I am trying to set the properties "push.data.poll.frequency" and "temp.csv.file.path". Earlier, these properties existed in the "mule-app.properties" file.
So, My question is, How do I set the properties loaded from the database to the flow. Please keep in mind that I have already loaded the properties from the database as described in the link above. I just want to be able to set these properties to the flow rather than taking them from the mule-app.properties.
EDIT: To add some more information,
I am using a class with #Configuration annotation. The class as described in the link above, loads the properties from the database. Below is the source code.
#Configuration(name="databasePropertiesProvider")
#Component
public class DatabasePropertiesProvider {
#Autowired(required=true)
private MyService myService;
#Bean
public Properties getProperties() throws Exception {
Properties properties = new Properties();
// get properties from the database
Map<String,String> propertiesMap = myService.getMuleAppPropertiesFromDB();
if(null != propertiesMap && !CollectionUtils.isEmpty(propertiesMap))
properties.putAll(propertiesMap);
return properties;
}
#Bean
public static PropertySourcesPlaceholderConfigurer placeHolderConfigurer() {
return new PropertySourcesPlaceholderConfigurer();
}}
But this class runs after the app is initialized. Previously, I had configured the PropertySourcesPlaceholderConfigurer in the xml config with the factory-bean as the DatabasePropertiesProvider class. But since DatabasePropertiesProvider has a dependency on MyService class, and the dependency was not getting resolved due to MyService bean not initializing in the container before the property config, I had to make some changes to DatabasePropertiesProvider(the version above) so that this runs after the app initialization.
But now, the problem is that I am unable to access those properties that are loaded from the database.
UPDATE 2: I found a solution. Apparently I was trying to autowire the #Service MyService in the databasePropertiesProvider class. The autowiring was failing with null due to which I made some more modifications to the databasePropertiesProvider class so that It runs after the app is initialized.
Now when I look at it, I realized that I dont need to connect to the database through all the service and repository layers. I moved the query execution code from the repository class to the databasePropertiesProvider class and now the properties are loaded during initialization time and the flows can get the properties without making any changes.
Thanks for all your help guys. Made me do a lot of thinking.
Regards,
Zulfiqar
I found a solution. Apparently I was trying to autowire the #Service MyService in the databasePropertiesProvider class. The autowiring was failing with null due to which I made some more modifications to the databasePropertiesProvider class so that It runs after the app is initialized.
Now when I look at it, I realized that I dont need to connect to the database through all the service and repository layers. I moved the query execution code from the repository class to the databasePropertiesProvider class and now the properties are loaded during initialization time and the flows can get the properties without making any changes.
The whole code looks like this
XML Config:-
<bean class="net.intigral.reports.provider.properties.DatabasePropertiesProvider" id="databasePropertiesProvider">
<property name="entityManager" ref="entityManager" />
</bean>
<bean class="org.springframework.context.support.PropertySourcesPlaceholderConfigurer">
<property name="properties">
<bean factory-bean="databasePropertiesProvider" factory-method="getProperties" />
</property>
</bean>
Java Code:-
public class DatabasePropertiesProvider {
EntityManager entityManager;
public Properties getProperties() throws Exception {
Properties properties = new Properties();
// get properties from the database
Map<String,String> propertiesMap = getMuleAppPropertiesFromDB();
if(null != propertiesMap && !CollectionUtilsIntg.isEmpty(propertiesMap))
properties.putAll(propertiesMap);
return properties;
}
public EntityManager getEntityManager() {
return entityManager;
}
public void setEntityManager(EntityManager entityManager) {
this.entityManager = entityManager;
}
#SuppressWarnings("unchecked")
private Map<String,String> getMuleAppPropertiesFromDB() {
Map<String,String> collect = null;
String query = "select key, value from MuleAppProps muleAppProps";
List<Object[]> results = entityManager.createQuery(query).getResultList();
if (CollectionUtilsIntg.isNotEmpty(results)) {
collect = results.stream().collect(Collectors.toMap(o -> (String)o[0], o -> (String)o[1]));
}
return collect;
}}
Now, I am able to load the properties the same way I used to load from mule-app.properties in the FLOWs.
Let your db contains following properties values with key/value pair as below :-
A simple example like below you can refer to read values from Database:-
<spring:beans>
<spring:bean id="dataSource" name="myCon" class="org.enhydra.jdbc.standard.StandardDataSource">
<spring:property name="url" value="jdbc:sqlserver://YourIpAddress\\SQLEXPRESS:1433;databaseName=YourDB;user=sa;password=yourDBPassword" />
<spring:property name="driverName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver" />
</spring:bean>
<!-- Required to connect to datasource -->
<spring:bean name="PropertyPlaceholderConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<spring:property name="properties" ref="CommonsConfigurationFactoryBean" />
</spring:bean>
<spring:bean name="CommonsConfigurationFactoryBean"
class="org.springmodules.commons.configuration.CommonsConfigurationFactoryBean">
<spring:constructor-arg ref="DatabaseConfiguration" />
</spring:bean>
<spring:bean name="DatabaseConfiguration" class="org.apache.commons.configuration.DatabaseConfiguration">
<spring:constructor-arg type="javax.sql.DataSource" ref="dataSource" />
<spring:constructor-arg index="1" value="YourTableName" />
<spring:constructor-arg index="2" value="Key" />
<spring:constructor-arg index="3" value="Value" />
</spring:bean>
</spring:beans>
<db:generic-config name="Database_Configuration" dataSource-ref="dataSource" doc:name="Generic Database Configuration" />
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration" />
<flow name="firstflow" processingStrategy="synchronous">
<http:listener config-ref="HTTP_Listener_Configuration" path="/test" doc:name="HTTP" />
<set-payload value="File name ${file.name} File path ${file.path}" doc:name="Set Payload" />
</flow>
You need to add commons-configuration.jar, spring.jar and spring-modules-jakarta-commons.jar in your classpath
If you want to access properties values in Java class you can inject it using Spring property in init-method of Spring bean.
refer:- http://www.codeproject.com/Articles/28893/Loading-Application-Properties-from-a-Database

Getting the Wrong Routing Key for AMQP message using RabbitMQ and Apache Camel

I am having an hard time tracking down an issue with my Camel routes. From what Ive been reading, it seems it may be something to do with my routing key header information getting screwed up, but Im not sure how to resolve this. Also, this is a Java OSGi project if thats important, but all the Camel stuffs are currently implemented in XML. Any help is appreciated.
Here's what Im trying to do:
<!-- The first route creates an object with some info in it and drop it on a rabbitmq
exchange called message.added -->
<route id="directIn">
<from uri="direct:in" />
<bean ref="connector" method="handleIncoming" />
<marshal id="marshal-one" ref="firstObject" />
<to uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.added" />
</route>
<!-- This route listens to message.added, processes the data, creates a new object, and
drops it on a different rabbitmq exchange called message.rest -->
<route id="addedOne">
<from uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.added" />
<unmarshal id="unmarshal-one" ref="firstObject" />
<bean ref="connector" method="processAndConvert" />
<marshal id="marshal-out" ref="secondObject" />
<to uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.rest" />
</route>
This code seems to have been working fine. The problem arises when we add a new bundle, which is also listening to the message.added rabbitmq message (which I believe you are allowed to do?)
<!-- This route is in a different bundle, also listening to message.added, processing
the data and creating the same object (from a common bundle) and dropping it
on the same rabbitmq exchange as before -->
<route id="addedTwo">
<from uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.added" />
<unmarshal id="unmarshal-two" ref="firstObject" />
<bean ref="someService" method="processUpdate" />
<marshal id="marshal-out2" ref="secondObject" />
<to uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.rest" />
</route>
This route fails with an error message that it is trying to unmarshal a secondObject into a firstObject in the addedTwo route.
ERROR | RabbitMQConsumer | DefaultErrorHandler ....
caught: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException:
Unrecognized field "secondObject" (class com.pointer.dangling.FirstObject["secondObject"])
Message History:
RouteId ProcessorId Processor
[addedTwo] [addedTwo] [ ]
[addedTwo] [unmarshal-two] [unmarshal[ref:firstObject] ]
Exchange:
Headers: {
CamelRedelivered=false,
rabbitmq.DELIVERY_TAG=1,
rabbitmq.EXCHANGE_NAME=me.ex,
rabbitmq.ROUTING_KEY=message.added
}
Body: {
"secondObject": {
// a bunch of fields for secondObject
}
}
It appears that a 'secondObject' is, at some point, making its way onto the 'message.added' exchange and being picked up by the 'addedTwo' route which tries to marshal it into a 'firstObject'. But Im not explicitly telling it to do that anywhere in the code - any ideas?
In the RabbitMQ Camel component, the routingKey endpoint option only works for Consumers (<from />).
Producers must set their routing key explicitly as a message header before the <to />. Your addedOne route should look like this:
<route id="addedOne">
<from uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true&routingKey=message.added" />
<unmarshal id="unmarshal-one" ref="firstObject" />
<bean ref="connector" method="processAndConvert" />
<marshal id="marshal-out" ref="secondObject" />
<setHeader headerName="rabbitmq.ROUTING_KEY">
<constant>message.rest</constant>
</setHeader>
<to uri="rabbitmq://localhost:5672/me.ex?exchangeType=topic&durable=false&autoDelete=true" />
</route>
See https://camel.apache.org/rabbitmq.html for more info.

How to invoke a webservice with dataFormat as PAYLOAD

I'm using camel 2.16.0
Created a camel route to invoke a web service, dataFormat as MESSAGE and i get the response normally.
and this route is invoked using ProducerTemlate
//payloadXml is a string which contains SOAP Request Message.
Object response = producerTemplate.requestBody("direct:invokeWS", payloadXml);
<route id="my_Sample_Camel_Route_with_CXF">
<from uri="direct:invokeWS" />
<to uri="cxf://http://localhost:8111/camel_MQ/TestService?wsdlURL=http://localhost:8111/camel_MQ/TestService?wsdl&serviceName={http://www.test.org/interface/test/ws}camel_MQ-ws&portName={http://www.test.org/interface/test}TestEndpoint&dataFormat=MESSAGE" />
<log message="------------->> ${body}" />
</route>
But once i change the dataFormat to "PAYLOAD"
I get exception.
Caused by: java.lang.IllegalArgumentException: The PayLoad elements cannot fit with the message parts of the BindingOperation. Please check the BindingOperation and PayLoadMessage.
at org.apache.camel.component.cxf.CxfEndpoint$CamelCxfClientImpl.setParameters(CxfEndpoint.java:1171)
Tried creating CxfPayload and then sent that to producerTeamplate while invoking the WS, but still the same Exception,
Finally I'm able to invoke WS using dataFormat as payload.
created CxfPayload object and added SOAP Headers and Body to it.
But still i was getting the same exception "The PayLoad elements cannot fit with the message parts of the BindingOperation"
Then I added defaultOperationName & defaultOperationNamespace headers while invoking the webservice as shown below.
<to uri="cxf:bean:camel_MQ_MQ-ws?dataFormat=PAYLOAD&defaultOperationName=TestService&defaultOperationNamespace=http://www.camel_MQ.org/interface&loggingFeatureEnabled=true" />
hope this helps ;-)

Mule ESB:add <service-overrides messageReceiver="myReceiver"> in sftp connector doesn't work

I use mule to process the files from sftp, but I need to process the file in order(the sequence is order by name of file).
For example:
there are 3 files in sftp:
1.txt,2.txt,3.txt
I need to process these three files one by one and have to in the order of file's name.(first to process 1.txt then 2.txt at last 3.txt)
I use service-overrides in connector and extend the SftpMessageReceiver and override the poll() method ,in the method I make these files's name by order. But it doesn't work.Who can tell me where is wrong.
the class I write
public class NewSftpMessageReceiver extends SftpMessageReceiver {
private SftpReceiverRequesterUtil sftpRRUtil = null;
public NewSftpMessageReceiver(SftpConnector connector,
FlowConstruct flow,
InboundEndpoint endpoint,
long frequency) throws
CreateException {
super(connector, flow, endpoint);
this.setFrequency(frequency);
sftpRRUtil = new SftpReceiverRequesterUtil(endpoint);
}
public NewSftpMessageReceiver(SftpConnector connector, FlowConstruct flow, InboundEndpoint endpoint)
throws
CreateException {
super(connector, flow, endpoint);
sftpRRUtil = new SftpReceiverRequesterUtil(endpoint);
}
#Override
public void poll() throws
Exception {
try {
String[] files = sftpRRUtil.getAvailableFiles(false);
//do something here.....
for (String file : files) {
if (getLifecycleState().isStopping()) {
break;
}
routeFile(file);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
the config file like below
<sftp:connector name="incoming-mandate-connector"
archiveDir="${esb.archive.path}/mandate/incoming"
useTempFileTimestampSuffix="true"
autoDelete="true"
identityFile="${esb.incoming.sftp.identityFile}"
passphrase="${esb.incoming.sftp.passphrase}"
sizeCheckWaitTime="${esb.incoming.sftp.sizeCheckWaitTime}">
<service-overrides messageReceiver="com.xxy.NewSftpMessageReceiver"/>
</sftp:connector>
<sftp:endpoint name="incoming-mandate-ep"
address="${esb.incoming.connector}"
connector-ref="incoming-mandate-connector"
keepFileOnError="true" encoding="${esb.encoding}">
<file:filename-wildcard-filter pattern="${esb.incoming.filePattern}"/>
</sftp:endpoint>
the flow like blew
<flow name="process">
<quartz:inbound-endpoint name="quartz-endpoint" cronExpression="${esb.cronExpression}" jobName="incoming-mandate-job">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="incoming-mandate-ep"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<processor ref="mandate-processor"/>
<transformer ref="mandate-transformer"/>
<mulexml:jaxb-object-to-xml-transformer jaxbContext-ref="jaxb" name="mandate-jaxb-transformer"/>
<byte-array-to-string-transformer mimeType="application/xml"/>
<message-properties-transformer scope="outbound">
<add-message-property key="${esb.username}" value="${esb.password}"/>
</message-properties-transformer>
<outbound-endpoint ref= "mandate-service-ep"/>
</flow>
The reason why your message receiver is not being called is because you have a Quartz endpoint that calls the SFTP transport, so in fact you are using a message requester, not a receiver.
You would need to create a custom message requester instead of a receiver and also a custom requester factory (that creates instances of the requester)
You can configure it as follows:
<sftp:connector name="sftpConnector">
<spring:property name="serviceOverrides">
<spring:map>
<spring:entry key="requester.factory" value="your.package.YourCustomSftpRequesterFactory"/>
</spring:map>
</spring:property>
</ftp:connector>

URL Encoding a Camel Exchange Body

I need help with this question.
I'm using the camel-http component as shown here but I'm having trouble because the body I'm sending has unescaped ampersands. This is causing the query string on the receiving server to break the post into multiple post parameters.
I know I could create compiled routes in java, but I must use the spring xml dialect so that new routes may be create/changed in the config files without a recompile.
So, in short, I'd like to URL Encode the ${body} property on my route using the spring dialect as shown in the (obviously invalid) pseudocode below.
<setBody inheritErrorHandler="true" id="setBody2">
<simple>name=<urlencode>${body}</urlencode></simple>
</setBody>
Ok, I bit the bullet. I created a java POJO
package com.wufoo.camel;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import org.apache.log4j.Logger;
public class PayloadEncoder {
public String getEncodedBody(String body) throws UnsupportedEncodingException {
Logger mylogger = Logger.getLogger("log4j.logger.org.apache.camel");
mylogger.info("Appending payload and URL Encoding");
String encodedBody = new StringBuffer()
.append("payload=")
.append(URLEncoder.encode(body, "UTF-8")).toString();
return encodedBody;
}
}
Then injected it into the context
<bean id="payloadEncoder" class="com.wufoo.camel.PayloadEncoder" />
And finally used a transform to encode the body
<transform>
<method bean="payloadEncoder" method="getEncodedBody"/>
</transform>
That works. If anyone can tell me what's wrong with this approach, please let me know.
You can also use groovy language, like this:
<?xml version="1.0" encoding="UTF-8"?>
<blueprint
xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="
http://www.osgi.org/xmlns/blueprint/v1.0.0
http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
<camelContext xmlns="http://camel.apache.org/schema/blueprint">
<route>
<from uri="file:camel/input"/>
<log message="Moving ${file:name} to the output directory"/>
<setBody>
<groovy>
"name=" + URLEncoder.encode(request.getBody(String.class));
</groovy>
</setBody>
<to uri="file:camel/output"/>
</route>
</camelContext>
</blueprint>