In MuleStudio a simple rename script deletes files after a while. This is how the script looks like:
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns:ftp="http://www.mulesoft.org/schema/mule/ftp" xmlns:file="http://www.mulesoft.org/schema/mule/file" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans" version="CE-3.4.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/ftp http://www.mulesoft.org/schema/mule/ftp/current/mule-ftp.xsd">
<flow name="ftp_ping_pong_testFlow1" doc:name="ftp_ping_pong_testFlow1" initialState="started" processingStrategy="queued-asynchronous">
<file:inbound-endpoint path="C:\ftp_ping_pong_test\src\test\in\" responseTimeout="10000" doc:name="File" fileAge="1000" pollingFrequency="1200">
<file:filename-regex-filter pattern="^.*\.csv$" caseSensitive="true"/>
</file:inbound-endpoint>
<file:outbound-endpoint path="C:\ftp_ping_pong_test\src\test\in\" outputPattern="#[message.inboundProperties['originalFilename']].temp" responseTimeout="10000" doc:name="File"/>
</flow>
<flow name="ftp_ping_pong_testFlow2" doc:name="ftp_ping_pong_testFlow2">
<file:inbound-endpoint path="C:\ftp_ping_pong_test\src\test\in\" responseTimeout="10000" doc:name="File" fileAge="1000" pollingFrequency="1200">
<file:filename-regex-filter pattern="^.*\.temp$" caseSensitive="true"/>
</file:inbound-endpoint>
<file:outbound-endpoint path="C:\ftp_ping_pong_test\src\test\in\" outputPattern="#[message.inboundProperties['originalFilename'].replace('.temp','')]" responseTimeout="10000" doc:name="File"/>
</flow>
</mule>
The script does nothing else but renames the file, than names it back. It makes no difference if I raise the fileAge or if I change the processing strategy or if I run it in MuleStudio or Mule Standalone. What is strange that it works for ~300 iterations but then it gets confused.
The error log what I get looks like:
WARN 2013-12-19 12:43:15,815 [Finalizer] org.mule.transport.file.FileMessageReceiver: Failure trying to remove file C:\ftp_ping_pong_test\src\test\in\145278.csv.temp from list of files under processing
I have already lost a week with this issue, so any help would be much appreciated :)
Update:
I created two different apps. Here they are:
<?xml version="1.0" encoding="UTF-8"?>
<mule Links removed by Stackoverflow">
<file:connector name="File_Connector_CSV" autoDelete="true" streaming="false" validateConnections="true" doc:name="File" readFromDirectory="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\in" writeToDirectory="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\out"/>
<flow name="ftp_ping_pong_testFlow1" doc:name="ftp_ping_pong_testFlow1" initialState="started" processingStrategy="synchronous">
<file:inbound-endpoint path="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\in\" responseTimeout="10000" doc:name="File" fileAge="1000" pollingFrequency="1200" connector-ref="File_Connector_CSV" >
<file:filename-regex-filter pattern="^.*\.csv$" caseSensitive="true"/>
</file:inbound-endpoint>
<file:outbound-endpoint outputPattern="#[message.inboundProperties['originalFilename']].temp" responseTimeout="10000" doc:name="File" path="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\out"/>
</flow>
</mule>
And the second:
<?xml version="1.0" encoding="UTF-8"?>
<mule links removed by Stackoverflow">
<file:connector name="File" writeToDirectory="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\in" readFromDirectory="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\out" autoDelete="true" streaming="false" validateConnections="true" doc:name="File"/>
<flow name="ftp_ping_pong_player2Flow1" doc:name="ftp_ping_pong_player2Flow1">
<file:inbound-endpoint path="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\out" responseTimeout="10000" doc:name="File" connector-ref="File">
<file:filename-regex-filter pattern="^.*\.temp$" caseSensitive="true"/>
</file:inbound-endpoint>
<file:outbound-endpoint path="C:\Users\gabor.bodo\MuleStudio\workspace\ftp_ping_pong_test\src\test\in\" outputPattern="#[message.inboundProperties['originalFilename'].replace('.temp','')]" responseTimeout="10000" doc:name="File" connector-ref="File"/>
</flow>
</mule>
I ran the test with 7 test files, and as usual the files started to disappear. From the combine logs, here is the last appearance of one of the files:
INFO 2013-12-20 17:58:13,843 [[ping_pong_player1].File_Connector_CSV.receiver.01] org.mule.transport.file.FileMessageReceiver: Lock obtained on file: C:\ftp_ping_pong_test\src\test\in\09.csv
INFO 2013-12-20 17:58:14,519 [[ping_pong_player1].File_Connector_CSV.receiver.01] org.mule.transport.file.FileConnector: Writing file to: C:\ftp_ping_pong_test\src\test\out\09.csv.temp
INFO 2013-12-20 17:58:14,812 [[ping_pong_player2].File.receiver.01] org.mule.transport.file.FileMessageReceiver: Lock obtained on file: C:\ftp_ping_pong_test\src\test\out\09.csv.temp
INFO 2013-12-20 17:58:15,437 [[ping_pong_player2].File.dispatcher.402] org.mule.transport.file.FileConnector: Writing file to: C:\ftp_ping_pong_test\src\test\in\09.csv
After these log entries this file simply disappeared without any no further trace in logs.
I have observed that even if in MuleStudio there is a default value defined for File Age, the value is not represented in the XML file, this is why Player2, has no File Age defined.
I can about a possible scenario:
- Player1, reads the csv file and creates the temp. Player 2 reads the temp and writes back the csv. Player1 deletes the csv, believing it is still the original file. In this case Player1 is too slow, lets Player2 intervene and this can happen if Player2 has no File Age. - I have to retest this with File Age value settings accordingly.But it does not explain why happens this very rarely.
Thanks,
Gabor
Update
I know your feeling. When you add the File Connector, uncheck Auto Delete in the General tab:
Finally, I managed to figure out what was the problem after countless of experiments.
It seems that when mule is launching an application the File Age property will be checked only once, and not for each flow separately (maybe it is normal but I did not expect this behaviour). Even with different flows, it doesn't matter what is the File Age property as long as it is less than the standard kick time of Mule (says 5000 ms, but actually it is 50000 ms - someone added one zero in plus somewhere).
So what happened, that Player1 and Player2 were always trying to read, copy and delete all the files in the same time. It was just a question of when until the scenario described by me above happened.
The solution now works well, by making two separate applications with big enough FileAge (Attention that it is not enough that you have a FileAge defined in MuleStudio, it has to be in the XML as well!).
Hope it saves time for someone and thanks for all the good intention!
Related
I've got a few .properties files containing some data. How do I write a specific value of a property from one of the property files to a variable?
There are the following three approaches to perform on this problem statement.
1.Reading a properties file using ${Key} expression
2.Reading a properties file using ![p[‘Key’]] expression
3.Reading a properties file using p() function from DataWeave
While according to your need, I have used the first approach. Please go through the following code. I think this will resolve your problem.
<http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="8081" doc:name="HTTP Listener Configuration"/>
<context:property-placeholder location="propread.properties"/>
<flow name="propreadFlow">
<http:listener config-ref="HTTP_Listener_Configuration" path="/propread" doc:name="HTTP"/>
<logger message=""Flow Started"" level="INFO" doc:name="Logger"/>
<set-variable variableName="prop" value="${prop.read}" doc:name="Variable"/>
<logger message="#[flowVars.prop]" level="INFO" doc:name="Logger"/>
</flow>
</mule>
Please create one property file under src/main/resources with the name propread.properties and copy this ( prop.read=HelloMule ) into the file and run the above code. You will get the solution.
We process XML-Files from the Filesystem and once they've been process, they will be moved to another directory. Another system monitors this output folder and takes all newly created files for further process.
If we define the moveToPattern and moveToDirecory properties, the input file will be first copied to destinations and then deleted from input folder. This works properly on small files but takes longer on huge files which causes to incomplete file on output folder for some seconds.
Our solution is to let the file be copied with .tmp extension and then rename it to the final name and delete the input file.
So the Flow looks like below:
Input-File --> Processing --> Output-File with .tmp --> Rename --> Delete Input-File
The Problem is through using this custom-transformer for renaming the file after the file:endpoint the flow remains active and all Messages won't be garbage collected and causes a OutOfMemory Exception.
A Memory Dump shows a huge number of DefaultMuleMessage object which proofs my assumption.
has anybody an Idea to resolve this Problem?
<file:connector name="inputFileConnector" autoDelete="false" doc:name="File" streaming="true" validateConnections="true" />
<file:connector name="temporaryFileOutputConnectorConfig" outputPattern="#[message.outboundProperties['transportFileName']].tmp" autoDelete="false" streaming="false" validateConnections="true" doc:name="File" />
<file:endpoint path="${outputDir}/${queuePrefix}#[message.outboundProperties.'queue_number' ]/#[message.outboundProperties.'approvalSubDir']" name="OutputConnectorEndpoint" responseTimeout="10000" doc:name="File"/>
<flow name="Queue_Flow" processingStrategy="synchronous" initialState="stopped" >
<file:inbound-endpoint path="${inputDir}" responseTimeout="10000" pollingFrequency="5000" connector-ref="inputFileConnector" doc:name="File">
.
.
.
<flow-ref name="OutStrategyFile" doc:name="Out Strategy File"/>
</flow>
<sub-flow name="OutStrategyFile">
<file:outbound-endpoint connector-ref="temporaryFileOutputConnectorConfig" responseTimeout="10000" ref="OutputConnectorEndpoint" doc:name="Write temporary file"/>
<custom-transformer class="com.fileutil.FileRenamer" doc:name="Rename file to final location">
<spring:property name="fileUtils" ref="fileUtils"/>
</custom-transformer>
<custom-transformer class="com.fileutil.DeleteOriginalInputMessage" doc:name="Delete Original input message"/>
</sub-flow>
I have a little question about mule xml config file,while I try simple mule app that using logger and expression.The config file be like this :
<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.4.1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http
http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd">
<flow name="basic_tutorialFlow1" doc:name="basic_tutorialFlow1">
<http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="8084" doc:name="HTTP"/>
<expression-filter expression="#[message.payload != '/favicon.ico']" doc:name="Expression"></expression-filter>
<logger level="INFO" doc:name="Logger" message="Current payload is #[message.payload]"/>
<set-payload doc:name="Set Payload" value="#['Hello, ' + message.payload + '. Today is ' + server.dateTime.format('dd/MM/yy') + '.' ]"/>
</flow>
I have error on expression-filter and logger. The error shown like this :
cvc-complex-type.2.4.a: Invalid content was found starting with element 'expression-filter'
The program can running but I don't feel right when the project still have error.I want to know how to fix this.
I check http://www.mulesoft.org/schema/mule/ but don't see expression-filter or logger.
my editor is STS 3.4.0 and mule version is 3.4 ,Thank you.
I'd recommend changing your schema locations to name your explicit version, e.g.
xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core
http://www.mulesoft.org/schema/mule/core/3.4/mule.xsd
http://www.mulesoft.org/schema/mule/http
http://www.mulesoft.org/schema/mule/http/3.4/mule-http.xsd"
and also the xmlns statements, similarly. I've experienced this with mule 3.2.x, where certain XML attributes would fail to resolve in Mule IDE, although they compiled just fine. Changing from "current" to my specific version often corrected these resolution errors (although not always).
In general, it's usually safe to ignore these resolution failures if compilation and execution succeeds, but I agree with your general assertion that it's not a good feeling. Try being explicit with your schema definition, as I searched the core mule.xsd and both logger and expression-filter are properly defined in that file, so that should resolve your resolution errors.
I have sample Munit Test case in My Mule Project. But the file always showing an error mark at <munit:config> tag at the start of the configuration file.
Note: But currently I'm able to execute my munit test cases.
The error is:
cvc-complex-type.2.4.a: Invalid content was found starting with
element 'munit:config'. One of
'{"http://www.mulesoft.org/schema/mule/core":annotations,
"http://www.mulesoft.org/schema/mule/core":description,
This is an old post, but I don't see an answer. I had the same issue:
I had
at the top of declarations. I deleted it since I also had
xmlns:core="http://www.mulesoft.org/schema/mule/core" below.
This fixed the error.
This error means that the munit jars are not in your classpath
check for duplicate names of files, or flow names.
even if the files are in different folders or packages.
make them unique no matter what !
To help anyone else and the future me who still face this issue in Anypoint Studio 6.5 and munit tools 3.9.0 we had to configure the munit configuration global element as follows
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:munit="http://www.mulesoft.org/schema/mule/munit" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tls="http://www.mulesoft.org/schema/mule/tls" xmlns:core="http://www.mulesoft.org/schema/mule/core" xmlns:mock="http://www.mulesoft.org/schema/mule/mock" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.mulesoft.org/schema/mule/munit http://www.mulesoft.org/schema/mule/munit/current/mule-munit.xsd
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/tls http://www.mulesoft.org/schema/mule/tls/current/mule-tls.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/mock http://www.mulesoft.org/schema/mule/mock/current/mule-mock.xsd">
<munit:config mock-connectors="false" mock-inbounds="false" name="munit"
doc:name="MUnit configuration"/>
<spring:beans>
<spring:import resource="classpath:insurer-sys-api.xml"/>
<spring:import resource="classpath:global.xml"/>
<spring:import resource="classpath:insurer-sys-api-claims.xml"/>
<spring:import resource="classpath:insurer-sys-api-policyValidation.xml"/>
<spring:import resource="classpath:insurer-ctp-sys-api-shared.xml"/>
<spring:import resource="classpath:insurer-ctp-sys-api-health.xml"/>
</spring:beans>
<http:request-config name="HTTPS_Request_Configuration" protocol="HTTPS"
host="localhost" port="8082" basePath="/api" doc:name="HTTP Request
Configuration">
<tls:context>
<tls:trust-store insecure="true"/>
</tls:context>
</http:request-config>
My requirement is to process a file at an inbound endpoint
Use Case 1- Failure in processing the file
If the file fails processing due to some exception like a validation error move the file to a separate directory different from the source. Delete the original file in the source directory
Use Case 2 -Successful processing
If the file processing is successful done , let the file remain in the source directory.
I tried the following , the file is moved to failed directory but the source file is not deleted as required in Use Case 1.
<flow name="InValidFlow1" doc:name="InValidFlow1">
<file:inbound-endpoint responseTimeout="10000" doc:name="File" path="c:\filelanding\in2" pollingFrequency="100" connector-ref="input_2" moveToDirectory="c:\filelanding\in2" moveToPattern="#[header:originalFilename]-#[function:dateStamp]-Processed">
<file:filename-regex-filter pattern="(?!.*Processed|.*Failed)(.*)" caseSensitive="false"/>
</file:inbound-endpoint>
<test:component waitTime="20000"></test:component>
<custom-transformer class="com.XXX.XXX.service.ExceptionService" doc:name="Java"/>
<file:outbound-endpoint responseTimeout="10000" doc:name="File" path="c:\filelanding\out2" connector-ref="output_2"/>
<exception-strategy ref="fot_exception_strategy_single" doc:name="Reference Exception Strategy"/>
</flow>
<file:connector name="error_output_1" outputPattern="#[header:originalFilename]" doc:name="File"/>
<choice-exception-strategy name="fot_exception_strategy_single">
<catch-exception-strategy when="#[exception.causedBy(java.lang.RuntimeException)]" doc:name="Catch Exception Strategy">
<!-- Mark the status as failed-->
<file:outbound-endpoint connector-ref="error_output_1" responseTimeout="10000" doc:name="File" path="c:\filelanding\backup2" outputPattern="#[header:originalFilename]-#[function:dateStamp]-Failed" >
</file:outbound-endpoint>
</catch-exception-strategy>
</choice-exception-strategy>
Do i need to override any existing mule functionality to achieve this behavior. The source folder on failure shouldn't contain the file. The destination folder should have the file marked with "Failed" status.
For use case 1, in your exception strategy use the value in originalFilename in a MEL expression component to move the file to wherever you want.
You can use org.mule.util.FileUtils.moveFileWithCopyFallback() for that matter.
PS. #[header:originalFilename] is old style expression syntax, on Mule 3.3 and above use MEL #[message.inboundProperties.originalFilename].