I have a need to have three different scheduled jobs for picking up and transferring files to an SFTP server. Using examples I was able to create a single working flow. However, when I replicate that flow and adjust the configuration, I get an error complaining about 2 connectors matching protocol file.
It asks me to specify these, however, I have specified which endpoint should be used for each flow.
Does anyone have any ideas about what I'm doing wrong, or what Mule is looking for?
Flow definitions:
<file:endpoint name="partsDataConnector" path="${partsDataOriginFilePath}" pollingFrequency="5000" doc:name="partsDataFile"/>
<flow name="partsDataTransfer">
<quartz:inbound-endpoint jobName="partsDataTransfer"
repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="partsDataConnector"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<sftp:outbound-endpoint host="${destinationFileServerIp}" port="${destinationFileServerPort}"
path="${partsDataDestinationPath}" tempDir="${partsDataDestinationTempDir}"
user="${destinationFileServerUser}" password="${destinationFileServerPassword}"
outputPattern="#[header:originalFilename]" />
</flow>
<file:endpoint name="imageDataConnector" path="${imageDataOriginFilePath}" pollingFrequency="5000" doc:name="partsDataFile"/>
<flow name="imageDataTransfer">
<quartz:inbound-endpoint jobName="imageDataTransfer"
repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="imageDataConnector"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<sftp:outbound-endpoint host="${destinationFileServerIp}" port="${destinationFileServerPort}"
path="${imageDataDestinationPath}" tempDir="${imageDataDestinationTempDir}"
user="${destinationFileServerUser}" password="${destinationFileServerPassword}"
outputPattern="#[header:originalFilename]" />
</flow>
<file:endpoint name="customerDataConnector" path="${customerDataOriginFilePath}" pollingFrequency="5000" doc:name="partsDataFile"/>
<flow name="customerDataTransfer">
<quartz:inbound-endpoint jobName="customerDataTransfer"
repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="customerDataConnector" />
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<sftp:outbound-endpoint host="${destinationFileServerIp}" port="${destinationFileServerPort}"
path="${customerDataDestinationPath}" tempDir="${customerDataDestinationTempDir}"
user="${destinationFileServerUser}" password="${destinationFileServerPassword}"
outputPattern="#[header:originalFilename]" />
</flow>
Stacktrace:
2014-04-09 06:46:44,924 INFO [org.quartz.core.JobRunShell] Job mule.quartz://customerDataTransfer threw a JobExecutionException:
org.quartz.JobExecutionException: org.mule.transport.service.TransportFactoryException: There are at least 2 connectors matching protocol "file", so the connector to use must be specified on the endpoint using the 'connector' property/attribute. Connectors in your configuration that support "file" are: connector.file.mule.default, connector.file.mule.default.1, (java.lang.IllegalStateException) [See nested exception: org.mule.transport.service.TransportFactoryException: There are at least 2 connectors matching protocol "file", so the connector to use must be specified on the endpoint using the 'connector' property/attribute. Connectors in your configuration that support "file" are: connector.file.mule.default, connector.file.mule.default.1, (java.lang.IllegalStateException)]
at org.mule.transport.quartz.jobs.EndpointPollingJob.doExecute(EndpointPollingJob.java:176)
at org.mule.transport.quartz.jobs.AbstractJob.execute(AbstractJob.java:36)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:534)
The error message is asking you to declare explicit file:connector components. Now you have just file:endpoint components.
If you haven't defined a file connecotr. Try declaring one and add a connector-reference on each of the file endpoints.
<file:connector name="myFileConnector" ></file:connector>
Add the Connector reference on each of the file endpoints as below. Add the reference for all the three file endpoints.
<file:endpoint name="imageDataConnector" connector-ref="myFileConnector" path="${imageDataOriginFilePath}" pollingFrequency="5000" doc:name="partsDataFile"/>
Hope this helps.
Related
I'm trying to use mule inbound file connector with poll scope got error saying couldn't start endpoint. If I remove poll scope and use file connector with default polling and its working fine without any file path changes.
I was wondering why is Poll scope giving error? If file inbound connector not allowed to wrapped in poll scope, why anypoint studio showing poll scope in the wrap in option ?
I found similar question, but I didn't see detailed explanations.
Mule won't allow POLL message processor to read file using file Inbound?
Advance thanks for your response.
Use mule-module-requester https://github.com/mulesoft/mule-module-requester, together with the Poll Scheduler.
relevant posts: http://blogs.mulesoft.com/dev/mule-dev/introducing-the-mule-requester-module/
Another way is,
Set the FTP flow initialState="stopped", and let the poll scheduler start the flow. After the FTP processing, stop the flow again.
see sample code:
<ftp:connector name="FTP" pollingFrequency="1000"
validateConnections="true" moveToDirectory="/work/ftp/processed"
doc:name="FTP" />
<flow name="scheduleStartFTPFlow">
<poll doc:name="Poll">
<fixed-frequency-scheduler frequency="1"
timeUnit="MINUTES" />
<expression-component doc:name="START FTP FLOW"><![CDATA[if(app.registry.processFTPFlow.isStopped()){
app.registry.processFTPFlow.start();
}]]></expression-component>
</poll>
<logger message="Poll Logging: #[payload]" level="INFO"
doc:name="Logger" />
</flow>
<flow name="processFTPFlow" initialState="stopped">
<ftp:inbound-endpoint host="localhost" port="21"
path="/data/ftp" user="Sanjeet" password="sanjeet123" responseTimeout="10000"
doc:name="FTP" connector-ref="FTP" />
<logger message="Logging FTP #[payload]" level="INFO" doc:name="Logger" />
<expression-component doc:name="STOP FTP FLOW"><![CDATA[app.registry.processFTPFlow.stop();]]></expression-component>
</flow>
Please, provide SSCCE.
Based on your question you do not need Poll at all. File Connector already has this feature to check file periodically. Here is example which polls file every 0.123 seconds
<file:inbound-endpoint path="/tmp" responseTimeout="10000" doc:name="File" pollingFrequency="123"/>
My suggestion is to use the quartz connector beside the file connector and set the interval in the quartz connector. Or use the file connector itself having the poll frequency so no need to wrap the file in poll scope.
you can create a file endpoint in the global element section and then use mule requester to invoke that endpoint inside a poll scope.
<file:connector name="File1" autoDelete="true" streaming="true" validateConnections="true" doc:name="File"/>
<file:endpoint connector-ref="File1" name="File" responseTimeout="10000" doc:name="File" path="/"/>
<flow name="pocforloggingFlow1">
<poll doc:name="Poll">
<mulerequester:request resource="File" doc:name="Mule Requester"/>
</poll>
</flow>
I have a Mule flow which processes files in an inbound folder that are named AAA_[id_number].dat. However, I need to configure Mule to only process this file when a corresponding file named [id_number].dat is also available. The second file indicates that the first is ready for processing.
Is there a way I can configure an inbound endpoint in Mule to only start processing the AAA_ file when it's counterpart is present? The [id_number].dat file is purely for notification purposes, it should not be processed by Mule. The inbound endpoint has a regex filter to look for a file in the format AAA...
<!-- Mule Requester Config -->
<mulerequester:config name="muleRequesterConfig" doc:name="Mule Requester"/>
<!-- File Connectors -->
<file:connector name="inputTriggerConnector" pollingFrequency="100" doc:name="File"/>
<file:connector name="inputFileConnector" doc:name="File"/>
<file:connector name="outputFileConnector" doc:name="File"/>
<!-- File Endpoints -->
<file:endpoint name="inputFileEndpoint" path="src/test/input" responseTimeout="10000" doc:name="File">
<file:filename-regex-filter pattern="\d{6}.dat" caseSensitive="true"/>
</file:endpoint>
<!-- Trigger Flow -->
<flow name="triggerFlow" doc:name="triggerFlow">
<file:inbound-endpoint ref="inputFileEndpoint" connector-ref="inputTriggerConnector" pollingFrequency="1000" doc:name="Input Trigger"/>
<flow-ref name="mainFlow_StockB2C" doc:name="Flow Reference"/>
</flow>
<!-- Main Flow -->
<flow name="mainFlow" doc:name="mainFlow">
<mulerequester:request config-ref="muleRequesterConfig" resource="file://.../AAA_#[message.inboundProperties.originalFilename]?connector=inputFileConnector" timeout="6000" doc:name="Mule Requester"/>
<DO SOMETHING WITH AAA_ FILE>
<file:outbound-endpoint connector-ref="outputFileConnector" path="src/test/output" outputPattern="#[function:dateStamp].csv" responseTimeout="6000" doc:name="Output File"/>
</flow>
Why not filter set a file inbound filter for the [id_number].dat files (or one that excludes the AAA_ files), if those are only used for notification? Would make more sense in my opinion. You can then grab the file to be processed with the requester module inside the flow, based on the originalFileName property.
Just in case this might help someone who needs it, you can create a custom filter and include your own filtering logic in there. More details from this blog here
I've searched the forums for an answer to this. I found one almost identical question, though the answer left me still wondering.
An almost identical post was found here:
Mule - Schedule a flow to consume a web service
This poster stated the problem I am having very well.
I am also new to Mule and am trying to do the very same thing. I didnt realize I needed a payload since I thought the operation specification was essentially the payload.
Notice that I have a flow that includes cxf:jaxws-client and that client specifies a URL for the service and an operation "listTest".
What other payload do i need to specify in order to actually execute the service request?
I tried to add a dummy payload to the event generator (as suggested in referenced post), and that doesnt make a difference.
When I execute the mule application, and monitor the "test auditor web service" (using wireshark) i see four requests go out for the wsdl, and i see that wsdl returned, but i dont actually see the listTest operation getting invoked.
My flow is:
<http:connector name="HTTP_HTTPS" cookieSpec="netscape"
validateConnections="true" sendBufferSize="0" receiveBufferSize="0"
receiveBacklog="0" clientSoTimeout="10000" serverSoTimeout="10000"
socketSoLinger="0" doc:name="HTTP\HTTPS" />
<flow name="TestAuditorClient_CheckerFlow1" doc:name="TestAuditorClient_CheckerFlow1">
<quartz:outbound-endpoint jobName="GetTestList"
repeatInterval="10000" responseTimeout="10000" doc:name="Quartz">
<quartz:event-generator-job jobGroupName="GetTestList" />
</quartz:outbound-endpoint>
<cxf:jaxws-client operation="listTest"
clientClass="server.TestService_Service" port="TestServicePort"
wsdlLocation="http://192.168.66.7:8080/TestAuditorWebApp/TestService?wsdl"
doc:name="SOAPY" />
<outbound-endpoint
address="http://192.168.66.7:8080/TestAuditorWebApp/TestService"
doc:name="HTTP" />
<logger message="Received HTTP Response #[payload]" level="INFO"
doc:name="Logger" />
<!-- <outbound-endpoint exchange-pattern="request-response" address="http://192.168.66.17:8080/TestAuditorWebApp/TestService"
doc:name="HTTP"/> -->
<file:outbound-endpoint path="C:\tmp"
outputPattern="#[function:datestamp:dd-MM-yy]_#[function:systime].txt"
responseTimeout="10000" doc:name="Output File" />
</flow>
I am not only new to mule, but as well to stack overflow. So if there was a better way for me to ask a related question, please advise and excuse.
Thanks in advance.
Instead of Quartz, you can use a poll message processor to generate the instances of ListTest you need.
Assuming this class FQDN is server.TestService.ListTest (you didn't tell), the following should work:
<flow name="TestAuditorClient_CheckerFlow1">
<poll frequency="10000">
<set-payload value="#[lt=new server.TestService.ListTest(); lt.aField='aValue'; lt]" />
</poll>
...
Notice how you can set values on the POJO directly from the expression that creates it.
I have a mule flow as under
<flow name="flow1" doc:name="f1">
<file:inbound-endpoint path="C:\input" responseTimeout="10000"
doc:name="File" />
</flow>
<flow name="flow2" doc:name="f2">
<http:inbound-endpoint address="http://localhost:8080"
doc:name="HTTP" exchange-pattern="request-response" />
<flow-ref name="flow1" doc:name="Flow Reference" />
<file:outbound-endpoint path="C:\outputfile"
responseTimeout="10000" doc:name="File" />
</flow>
I am trying to move/upload multiple files from source to destination (can be anything e.g. FTP or File outbound etc..) by using the flow.
The reason for doing in this way is that I want to invoke the job from CLI(Command Line Interface) using CURL.
But it is not working....
Edited
I need to pick up some files(multiple files) from a particular folder located in my hard drive. And then move those to some outbound process which can be FTP site or some other hard drive location.
But this flow needs to be invoked from CLI.
Edited (Based on David's answer)
I now have the flow as under
<flow name="filePickupFlow" doc:name="flow1" initialState="stopped">
<file:inbound-endpoint path="C:\Input" responseTimeout="10000" doc:name="File"/>
<logger message="#[message.payloadAs(java.lang.String)]" level="ERROR" />
</flow>
<flow name="flow2" doc:name="flow2">
<http:inbound-endpoint address="http://localhost:8080/file-pickup/start" doc:name="HTTP" exchange-pattern="request-response"/>
<expression-component>
app.registry.filePickupFlow.start();
</expression-component>
<file:outbound-endpoint path="C:\outputfile" responseTimeout="10000" doc:name="File"/>
</flow>
I am getting couple of problems
a) I am getting an error that - Attribute initialState is not defined as a valid property of flow
However, if I remove that attribute, the flow continues without waiting for "http://localhost:8080/file-pickup/start" to fire up.
b) The files are not moved to the destination folder
So how can I do so?
You can't reference a flow that has an inbound endpoint in it because such a flow is already active and consuming events from its inbound endpoint so you can't invoke it on demand.
The following, tested on Mule 3.3.1, shows how to start a "file pickup flow" on demand from an HTTP request.
<flow name="filePickupFlow" initialState="stopped">
<file:inbound-endpoint path="///tmp/mule/input" />
<!-- Do something with the file: here we just log its content -->
<logger message="#[message.payloadAs(java.lang.String)]" level="ERROR" />
</flow>
<flow name="filePickupStarterFlow">
<http:inbound-endpoint address="http://localhost:8080/file-pickup/start"
exchange-pattern="request-response" />
<expression-component>
app.registry.filePickupFlow.start();
</expression-component>
<set-payload value="File Pickup successfully started" />
</flow>
HTTP GETting http://localhost:8080/file-pickup/start would then start the filePickupFlow, which in turn will process the files in /tmp/mule/input.
Note that it is up to you to configure the file:connector for what behavior it must have for files it processes, either deleting them or moving them to another directory are two main options.
I guess in this case a File inbound to read a file on demand will not be helpful.
Please try if the follwoing way.
<flow name="flow1" doc:name="f2">
<http:inbound-endpoint address="http://localhost:8080"
doc:name="HTTP" exchange-pattern="request-response" />
<component>
<spring-object bean="fileLoader"></spring-object>
</component>
<file:outbound-endpoint path="C:\outputfile"
responseTimeout="10000" doc:name="File" />
</flow>
So the Custom component will be a Class which reads the file from your specified location.
Hope this helps.
You can use Mule Requester for a clean solution. See the details in the blog entry Introducing the Mule Requester.
This is an extension of my previous question How to upload multiple files via REST over HTTP using Mule?. The requirement say that, on every Wednesday at 10AM the files has to be uploaded. Henceforth I need a scheduler for accomplishing this. And I found that the solution is "Quartz" inbound component with Cron Expression.
But how can I do so? Because I cannot have two "inbound-endpoint".(quartz and file) e.g.
<flow name="fileUploader" doc:name="fileUploader">
<quartz:inbound-endpoint
jobName="myServiceJob"
repeatInterval="5000"
cronExpression="0 0 10 ? * WED
doc:name="Quartz">
<quartz:event-generator-job/>
</quartz:inbound-endpoint>
<file:inbound-endpoint
path="C:\input"
pollingFrequency="5000" moveToDirectory="C:\movehere" doc:name="File"
responseTimeout="10000"/>
<object-to-byte-array-transformer doc:name="Object to Byte Array"/>
<file:outbound-endpoint
path="C:\outputfile"
responseTimeout="10000"
doc:name="File"/>
</flow>
If I run I get error
Exception in thread "main" org.mule.module.launcher.DeploymentInitException: SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'file:inbound-endpoint'.
So what is the change that I need to do?
Please help
Try this
<file:endpoint name="fileConnector" path="C:\input" pollingFrequency="5000" doc:name="File"/>
<flow name="fileUploader" doc:name="fileUploader">
<quartz:inbound-endpoint
jobName="myServiceJob"
repeatInterval="5000"
cronExpression="0 0 10 ? * WED"
doc:name="Quartz">
<quartz:endpoint-polling-job>
<quartz:job-endpoint ref="fileConnector"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
<file:outbound-endpoint
path="C:\outputfile"
responseTimeout="10000"
outputPattern="#[message.inboundProperties.originalFilename]"
doc:name="File"/>
</flow>
You have two options:
a. Replace the file inbound endpoint with a component that handles the File processing. It will be triggered by Quartz, pick up the file(s) from the folder and pass it to the outbound endpoint.
b. Don't use the Quartz endpoint and override org.mule.transport.file.FileMessageReceiver to implement your custom scheduling for polling files.
The first alternative is the easier one.
The following just a work round if you couldn't find the exact what you need.
1- You could have 2 flows instead of one, one to do the normal job with normal inbound and one for quartz:inbound-endpoint.
2- You could make your scheduler to put messages on a queue and have an other flow to pick up messages from that queue and process.
3- You could have a component with quartz:inbound-endpoint as suggested here http://blogs.mulesoft.org/using-quartz-to-trigger-a-service/
Hope one of the above helps.
Sorry couldn't put this as a comment as it is too long so here it is an example for <quartz:endpoint-polling-job> from Mule website:
<service name="testService5">
<description>
The endpoint polling Job will try and perform a 'request' on any Mule endpoint. If a result is received it will be handed off to this 'testService5' service for processing. The trigger will fire every 5 minutes starting at 2pm and ending at 2:55pm, every day. during this period the job will check the file directory /N/drop-data/in every 5 minutes to see if any event data is available. The request will timeout after 4 seconds if there are no files in the directory.
</description>
<inbound>
<quartz:inbound-endpoint name="qEP5" cronExpression="0 0/5 14 * * ?" jobName="job5"
connector-ref="quartzConnector1">
<quartz:endpoint-polling-job>
<quartz:job-endpoint address="file:///N/drop-data/in" timeout="4000"/>
</quartz:endpoint-polling-job>
</quartz:inbound-endpoint>
</inbound>
</service>
Hope the above helps