My WSO2 ESB has to copy large files (>100MB) from one directory to another. Earlier, when pollInterval was 5 seconds, a file wasn't completely copied. I set pollInterval to 180 seconds, and everything is ok now. But I need a guarantee, that file will be copied completely. There is a trailer at the end of file. So, I just need to check, is there a trailer at the end of copied file or not. Is it possible?
My esb_configuration.xml
<proxy name="proxy_name" transports="vfs" startOnLoad="true" trace="disable">
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.PollInterval">180</parameter>
<parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter>
<parameter name="transport.vfs.FileURI">vfs:file:///path/to/file</parameter>
<parameter name="transport.vfs.ActionAfterErrors">DELETE</parameter>
<parameter name="transport.vfs.FileNamePattern">^FILE_MASK$</parameter>
<parameter name="transport.vfs.ContentType">application/octet-stream</parameter>
<parameter name="transport.vfs.ActionAfterFailure">DELETE</parameter>
</proxy>
AFAIK, Poll Interval doesn't affect the files that are being transferred, it just acts as a scheduler for the transport receiver to poll the file URI location. Are you sure the file didn't get copied completely due to the poll interval being less?
I don't think there are any ways of checking the trailer record using the VFS parameters.
Maybe you can use transport.vfs.MaxRetryCount to retry in case of any failures, you can see the complete list of VFS service level parameters [1].
[1] - https://docs.wso2.com/display/ESB481/VFS+Transport#VFSTransport-parametersVFSservice-levelparameters
Related
I am making a consumer of rabbitMQ on WSO2 I need to send the message to one service one at the time I know that rabbitMQ has a setting call prefetch for that but I di not see the parameter on the wso2 documentation, any one knows other way to do this? Thanks in advance
My inbound endpoint looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<inboundEndpoint name="RabbitMQConsumerTesting" onError="SalesOrderQueueErrorSeq" protocol="rabbitmq" sequence="HandlerRabbitMessage" suspend="false" xmlns="http://ws.apache.org/ns/synapse">
<parameters>
<parameter name="sequential">true</parameter>
<parameter name="coordination">true</parameter>
<parameter name="rabbitmq.connection.factory">AMQPConnectionFactory</parameter>
<parameter name="rabbitmq.server.host.name">my_host</parameter>
<parameter name="rabbitmq.server.port">5672</parameter>
<parameter name="rabbitmq.server.user.name">my_user</parameter>
<parameter name="rabbitmq.server.password">my_password</parameter>
<parameter name="rabbitmq.queue.name">my_queue</parameter>
<parameter name="rabbitmq.exchange.name">my_queue</parameter>
<parameter name="rabbitmq.server.virtual.host">my_vhost</parameter>
</parameters>
</inboundEndpoint>
Basically, the prefetch value is used to specify how many messages are being sent at the same time in RabbitMQ. You can set the prefetch message count by using the below property in the latest WSO2MI-7.1.0 server[1].
rabbitmq.channel.consumer.qos
Please note that if a value is not set, 0 will be used as the default value.
I have used a message Processor to call an end Point and make a guaranteed delivery.
Sometimes it happens that a buggy message (incompatible data type with the end point WS) what causes ERROR HTTP SC: 500
all the follow up messages (this time correct ones) are stuck due to the first blocking message.
How to unblock the message processor?
how to delete it from the queue? possibly is there a way how to correct it?
This is very dangerous if it happens in the production.
thank you!
message processor
<?xml version="1.0" encoding="UTF-8"?>
<messageProcessor
class="org.apache.synapse.message.processor.impl.forwarder.ScheduledMessageForwardingProcessor"
messageStore="SAPMoveMaterialOriginalMsgStore"
name="SAPMoveMaterialMessageProcessor"
targetEndpoint="SAPmaterialMoveProxyService" xmlns="http://ws.apache.org/ns/synapse">
<parameter name="client.retry.interval">900000</parameter>
<parameter name="throttle">false</parameter>
<parameter name="max.delivery.attempts">50</parameter>
<parameter name="member.count">1</parameter>
<parameter name="message.processor.reply.sequence">handleSAPResponse</parameter>
<parameter name="message.processor.fault.sequence">fault</parameter>
<parameter name="max.delivery.drop">Enabled</parameter>
<parameter name="interval">1000</parameter>
<parameter name="is.active">true</parameter>
</messageProcessor>
The end point
<?xml version="1.0" encoding="UTF-8"?>
<endpoint name="zsap_material_move" xmlns="http://ws.apache.org/ns/synapse">
<address trace="disable" uri="http://r3cr1.echternach.ieeintern:8020/sap/bc/srt/rfc/sap/zsap_material_move">
<timeout>
<duration>60000</duration>
</timeout>
</address>
</endpoint>
Do not send the message directly to your endpoint, but to a proxy service where you define your own mediation and decide what kind of response to send back to the Message Processor
In the Message Forwarding processor, It you do not want to retry any messages which returns HTTP 500 you can specify "Non retry http status codes" to avoid. Also if you set the "Maximum redelivery attempts" property and set "Drop message after maximum delivery attempts" to true it will drop the message without blocking the processor. Also in the Fault sequence you can decide on the mediation login you need to perform on the Faulty message. You can refer the following screen for this.
I am using the following code in my proxy to connect ftp server to read file from.
<proxy name="SFTPVFSProxy" transports="vfs" startOnLoad="true" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<send/>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.MoveAfterProcess">vfs:sftp://username:password#ftpserver-host/outbox</parameter>
<parameter name="transport.vfs.FileURI">vfs:sftp://username:password#ftpserver-host/inbox</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:sftp://username:password#ftpserver-host/sent</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
But I am getting below error when I ran the above. VFS transport is eanbled in axis2.xml and I can see vfs transport in wso2 esb 4.9.0 admin console.
The system cannot infer the transport information from the vfs:sftp://user:password#sftpserverhost/in URL
SFTP uses SSH protocol to communicate with the server, so it always logs into the user's home directory and then it takes the path relative to the user's home directory.
You can add "sftpPathFromRoot" to WSO2 VFS transport URI as to get absolute path access as below.
<parameter name="transport.vfs.FileURI">sftp://[ username[: password]#] hostname[: port][ absolute-path]?sftpPathFromRoot=true</parameter>
Alternatively you can use symbolic link.You can find more details here.
Just starting out with InboundEndpoints in version 4.9.0 of the WSO2 ESB.
I would like to create an inbound endpoint that polls for incoming files, forwards this to a service and saves the service reply to a file.
I have read the available docs and examples and this is what I came up with.
<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"
name="testinbound"
sequence="TestIn"
onError="fault"
protocol="file"
suspend="false">
<parameters>
<parameter name="interval">1000</parameter>
<parameter name="coordination">true</parameter>
<parameter name="transport.vfs.ContentType">text/xml</parameter>
<parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
<parameter name="transport.vfs.AutoLockRelease">false</parameter>
<parameter name="transport.vfs.ActionAfterFailure">DELETE</parameter>
<parameter name="transport.vfs.CreateFolder">true</parameter>
<parameter name="sequential">true</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">//C:/Test/ST/in</parameter>
<parameter name="transport.vfs.MoveAfterFailure">//C:/Test/ST/failure</parameter>
<parameter name="transport.vfs.DistributedLock">true</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.vfs.MoveAfterProcess">//C:/Test/ST/original</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.FileSortAscending">true</parameter>
<parameter name="transport.vfs.FileSortAttribute">NONE</parameter>
<parameter name="transport.vfs.Build">false</parameter>
</parameters>
</inboundEndpoint>
where the "TestIn" sequence looks like:
<sequence xmlns="http://ws.apache.org/ns/synapse"
name="TestIn"
onError="fault"
statistics="enable">
<in>
<send>
<endpoint key="conf:/stock_quote_config/stock_quote_endpoint_reg"/>
</send>
</in>
</sequence>
Now I have a couple of questions about the behavior of this code, and how to achieve my goal.
How to get my hands on the reply of the service? I have tried to add an <out> mediator in the TestIn sequence. I got the following:
OutMediator Current message is a request - skipping child mediators
I also tried the transport.vfs.ReplyFileURI and transport.vfs.ReplyFileName parameters.
Strangely, these options are not available via the UI. I could specify them in the configuration source file, without any effect.
(These parameters are described in the documentation of the 4.9.0 version. See
https://docs.wso2.com/display/ESB490/VFS+Transport )
So how to save the reply? And how to redirect the reply?
I see that after the TestIn sequence the <main> sequence is executed by the ESB on the response path -- this is what I would like to redefine.
I get an error that the file could not be moved from the "in" to the "original" folder. (In fact, after execution the file is moved.)
[2015-11-20 13:36:55,145] ERROR - FilePollingConsumer Error moving
file : file:///c:/Test/ST/in/test.xml to //C:/Test/ST/original
org.apache.commons.vfs2.FileSystemException: Could not delete "file:///c:/Test/ST/in/test.xml".
at org.apache.commons.vfs2.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:849)
at org.apache.commons.vfs2.provider.AbstractFileObject.moveTo(AbstractFileObject.java:1139)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.moveOrDeleteAfterProcessing(FilePollingConsumer.java:842)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.directoryHandler(FilePollingConsumer.java:568)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.poll(FilePollingConsumer.java:214)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.execute(FilePollingConsumer.java:134)
at org.wso2.carbon.inbound.endpoint.protocol.file.FileTask.taskExecute(FileTask.java:45)
at org.wso2.carbon.inbound.endpoint.common.InboundTask.execute(InboundTask.java:44)
at org.wso2.carbon.mediation.ntask.NTaskAdapter.execute(NTaskAdapter.java:90)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not delete "c:\Test\ST\in\test.xml".
at org.apache.commons.vfs2.provider.local.LocalFile.doDelete(LocalFile.java:127)
at org.apache.commons.vfs2.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:838)
... 16 more
You can use Call mediator instead of send mediator. When the response is received, the mediation flow resumes from the next mediator in the sequence. So you can redirect or process the incoming response.
I have a client for a 3rd party webservice that had to be created using the Azis WSDL2Java tool. Every web request must be encrypted using an Encryption handler provided by the 3rd party and the incoming responses myst be Decrypted in the same format. This all works really well, except that I have the handlers specified in a client-config.wsdd in the src/main/resources of my jar. one of the parameters of the client-config is a username and one is the location to a key file. This will need to be deployed to many different environments where the user and key location will differ, and it's not going to be feasible to change the values each time then rebuild. Is there a way I can either pass in parameters to my wsdd or place the wsdd in the $JBOSS_HOME/server//conf folder etc? (I am using JBoss 5.1.0.GA)
The wsdd looks like this
<?xml version='1.0'?>
<deployment xmlns="http://xml.apache.org/axis/wsdd/" xmlns:java="http://xml.apache.org/axis/wsdd/providers/java">
<globalConfiguration>
<requestFlow>
<handler name="EncryptionHandler" type="java:com.3rdparty.services.util.EncryptionHandler" >
<parameter name="userName" value="myuser"/>
<parameter name="keyFile" value="/jboss-5.1.0.GA/server/<name>/conf/my.key"/>
<parameter name="passwordCallbackClass" value="com.3rdparty.security.client.PWCallback"/>
</handler>
</requestFlow>
<responseFlow>
<handler name="DecryptionHandler" type="java:com.3rdparty.services.util.DecryptionHandler" >
<parameter name="userName" value="myuser"/>
<parameter name="keyFile" value="/jboss-5.1.0.GA/server/<name>/conf/my.key"/>
</handler>
</responseFlow>
</globalConfiguration>
<transport name="http" pivot="java:org.apache.axis.transport.http.HTTPSender"/>
<transport name="local" pivot="java:org.apache.axis.transport.local.LocalSender"/>
<transport name="java" pivot="java:org.apache.axis.transport.java.JavaSender"/>
</deployment>
You can use the system property axis.ClientConfigFile to specify a particular directory of the client-config.wsdd file.