The system cannot infer the transport information from the vfs:sftp://user:password#sftpserverhost/in URL - wso2-esb

I am using the following code in my proxy to connect ftp server to read file from.
<proxy name="SFTPVFSProxy" transports="vfs" startOnLoad="true" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<send/>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.MoveAfterProcess">vfs:sftp://username:password#ftpserver-host/outbox</parameter>
<parameter name="transport.vfs.FileURI">vfs:sftp://username:password#ftpserver-host/inbox</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:sftp://username:password#ftpserver-host/sent</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
But I am getting below error when I ran the above. VFS transport is eanbled in axis2.xml and I can see vfs transport in wso2 esb 4.9.0 admin console.
The system cannot infer the transport information from the vfs:sftp://user:password#sftpserverhost/in URL

SFTP uses SSH protocol to communicate with the server, so it always logs into the user's home directory and then it takes the path relative to the user's home directory.
You can add "sftpPathFromRoot" to WSO2 VFS transport URI as to get absolute path access as below.
<parameter name="transport.vfs.FileURI">sftp://[ username[: password]#] hostname[: port][ absolute-path]?sftpPathFromRoot=true</parameter>
Alternatively you can use symbolic link.You can find more details here.

Related

WSO2EI 7.1.1 File Processing not working on centos server

I am trying this tutorial from this given link https://ei.docs.wso2.com/en/latest/micro-integrator/use-cases/tutorials/file-processing/
Here is the file proxy definition
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="daily-file-proxy" startOnLoad="true" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log level="full"/>
<clone>
<target sequence="daily-file-file-write-sequence"/>
<target sequence="daily-file-mail-notification-sequence"/>
<target sequence="daily-file-db-sequence"/>
</clone>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterProcess">$FILE:FILE_DAILY_PROCESS</parameter>
<parameter name="transport.vfs.FileURI">$FILE:FILE_DAILY_DOWNLOAD</parameter>
<parameter name="transport.vfs.MoveAfterFailure">$FILE:FILE_DAILY_FAILURE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
I also declared all the parameters in the file.properties as well.
This code worked properly from Integration Studio as I drop the txt file in the FILE:FILE_DAILY_PROCESS, it processed in 15 sec and put the data into database.
Then I packaged the project as car file and deployed to micro integerator wso2 in the centos server.
I checked the log and everything deployed properly without error (wso2carbon.log and wso2-error.log not show any error)
I am thinking that the wso2ei could not see the folder declaring
Here is the parameters declaring in file.properties
FILE_DAILY_DOWNLOAD=file:///root/file-processing/daily-download/202107
FILE_DAILY_PROCESS=file:///root/file-processing/daily-process
FILE_DAILY_PROCESS_URI=vfs:file:///root/file-processing/daily-process
FILE_DAILY_FAILURE=file:///root/file-processing/daily-failure
finally, I have figured out the issue. The wso2 could not see the specific folder under ssh user root. It must be under /home/wso2 so the VFS listener only see the folder available under /home/wso2 folder
So use the update settings
FILE_DAILY_DOWNLOAD=file:///home/wso2/file-processing/daily-download/202107
FILE_DAILY_PROCESS=file:///home/wos2/file-processing/daily-process
FILE_DAILY_PROCESS_URI=vfs:file:///home/wso2/file-processing/daily-process
FILE_DAILY_FAILURE=file:///home/wso2/file-processing/daily-failure

WSO2 proxy service to send files to SFTP server using SSH cannot authenticate

I am trying to send files to SFTP server (SSH) using a WSO2 proxy service, but it appears the "transport.vfs.SFTPIdentities" and "transport.vfs.SFTPIdentityPassPhrase" parameters are not being applied when using the "transport.vfs.ReplyFileURI" parameter.
The proxy is picking up the incoming file, and attempting to send it to the SFTP server, but authorization is failing. (com.jcraft.jsch.JSchException: Auth fail)
Note: I have an Inbound Endpoint configured using the same properties and values which successfully connects to the server and picks up incoming files without issue.
Has anyone configured a proxy service or endpoint to send files to an SFTP server using SSH? What am I missing?
The proxy configuration:
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="MyProxyService" startOnLoad="true" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<property expression="$trp:FILE_NAME" name="theFileName" scope="default" type="STRING"/>
<log level="custom">
<property expression="$ctx:theFileName" name="Processing file"/>
</log>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.vfs.SFTPUserDirIsRoot">false</parameter>
<parameter name="transport.vfs.ReplyFileURI">vfs:sftp://user#sftp.site/In?transport.vfs.AvoidPermissionCheck=true</parameter>
<parameter name="transport.PollInterval">30</parameter>
<parameter name="transport.vfs.FileURI">file://c:/TEMP</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.SFTPIdentities">/keys/my_private_key.ppk</parameter>
<parameter name="transport.vfs.ActionAfterFailure">NONE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file://c:/TEMP/Archive</parameter>
<parameter name="transport.vfs.SFTPIdentityPassPhrase">passphrase</parameter>
</proxy>
The relevant stack trace:
...
[2020-05-14 15:59:52,960] [EI-Core] INFO - SftpClientFactory [JSCH] Authentications that can continue: publickey,keyboard-interactive,password
[2020-05-14 15:59:52,960] [EI-Core] INFO - SftpClientFactory [JSCH] Next authentication method: publickey
[2020-05-14 15:59:52,960] [EI-Core] INFO - SftpClientFactory [JSCH] Disconnecting from sftp.site port 22
[2020-05-14 15:59:52,960] [EI-Core] ERROR - VFSTransportSender cannot resolve replyFile
org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://user#sftp.site/".
at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:108)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:155)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:119)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:88)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:747)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:626)
at org.apache.synapse.transport.vfs.VFSTransportSender.writeFile(VFSTransportSender.java:233)
at org.apache.synapse.transport.vfs.VFSTransportSender.sendMessage(VFSTransportSender.java:194)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:119)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp.site".
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.createConnection(SftpClientFactory.java:268)
at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:97)
... 12 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:512)
at com.jcraft.jsch.Session.connect(Session.java:183)
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.createConnection(SftpClientFactory.java:264)
... 13 more

How to handle VFS proxy error in WSO2 EI 6.4?

In VFS proxy, How I can handle errors? For example in below cases:
If VFS is not able to connect to SFTP server due to different reasons(like wrong password, directory not exist etc). I am able to see Warning in console log but I want to send a mail or want to do some other stuff if connection is not possible.
If the folder does not exist where we have to move file after reading.
I am able to handle error during processing in FaultSequence but that thing does not work for proxy. I tried to set onError attribute as well but seems proxy does not support that attribute.
Please let me know if someone handle such errors or I am missing something.
<proxy name="FileProxy" startOnLoad="false" statistics="enable" trace="enable" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target faultSequence="Global_Error_Handler">
<inSequence>
<log level="custom">
<property name="status=" value="File received."/>
</log>
</inSequence>
<outSequence/>
</target>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.PollInterval">10</parameter>
<parameter name="transport.vfs.FileURI">vfs:sftp://abc:***#127.0.0.1:22/test/ftp_in?transport.vfs.AvoidPermissionCheck=true</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:sftp://abc:***#127.0.0.1:22/test/ftp_failure</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.ReplyFileName">ReplyFileName.csv</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.dat</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///home/test/ftp_process</parameter>
</proxy>
Need to handle any error that occurs outside the <target> functionality of proxy for VFS.

How to process reply with InboundEndpoints in WSO2 ESB

Just starting out with InboundEndpoints in version 4.9.0 of the WSO2 ESB.
I would like to create an inbound endpoint that polls for incoming files, forwards this to a service and saves the service reply to a file.
I have read the available docs and examples and this is what I came up with.
<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse"
name="testinbound"
sequence="TestIn"
onError="fault"
protocol="file"
suspend="false">
<parameters>
<parameter name="interval">1000</parameter>
<parameter name="coordination">true</parameter>
<parameter name="transport.vfs.ContentType">text/xml</parameter>
<parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
<parameter name="transport.vfs.AutoLockRelease">false</parameter>
<parameter name="transport.vfs.ActionAfterFailure">DELETE</parameter>
<parameter name="transport.vfs.CreateFolder">true</parameter>
<parameter name="sequential">true</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">//C:/Test/ST/in</parameter>
<parameter name="transport.vfs.MoveAfterFailure">//C:/Test/ST/failure</parameter>
<parameter name="transport.vfs.DistributedLock">true</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.vfs.MoveAfterProcess">//C:/Test/ST/original</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.FileSortAscending">true</parameter>
<parameter name="transport.vfs.FileSortAttribute">NONE</parameter>
<parameter name="transport.vfs.Build">false</parameter>
</parameters>
</inboundEndpoint>
where the "TestIn" sequence looks like:
<sequence xmlns="http://ws.apache.org/ns/synapse"
name="TestIn"
onError="fault"
statistics="enable">
<in>
<send>
<endpoint key="conf:/stock_quote_config/stock_quote_endpoint_reg"/>
</send>
</in>
</sequence>
Now I have a couple of questions about the behavior of this code, and how to achieve my goal.
How to get my hands on the reply of the service? I have tried to add an <out> mediator in the TestIn sequence. I got the following:
OutMediator Current message is a request - skipping child mediators
I also tried the transport.vfs.ReplyFileURI and transport.vfs.ReplyFileName parameters.
Strangely, these options are not available via the UI. I could specify them in the configuration source file, without any effect.
(These parameters are described in the documentation of the 4.9.0 version. See
https://docs.wso2.com/display/ESB490/VFS+Transport )
So how to save the reply? And how to redirect the reply?
I see that after the TestIn sequence the <main> sequence is executed by the ESB on the response path -- this is what I would like to redefine.
I get an error that the file could not be moved from the "in" to the "original" folder. (In fact, after execution the file is moved.)
[2015-11-20 13:36:55,145] ERROR - FilePollingConsumer Error moving
file : file:///c:/Test/ST/in/test.xml to //C:/Test/ST/original
org.apache.commons.vfs2.FileSystemException: Could not delete "file:///c:/Test/ST/in/test.xml".
at org.apache.commons.vfs2.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:849)
at org.apache.commons.vfs2.provider.AbstractFileObject.moveTo(AbstractFileObject.java:1139)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.moveOrDeleteAfterProcessing(FilePollingConsumer.java:842)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.directoryHandler(FilePollingConsumer.java:568)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.poll(FilePollingConsumer.java:214)
at org.wso2.carbon.inbound.endpoint.protocol.file.FilePollingConsumer.execute(FilePollingConsumer.java:134)
at org.wso2.carbon.inbound.endpoint.protocol.file.FileTask.taskExecute(FileTask.java:45)
at org.wso2.carbon.inbound.endpoint.common.InboundTask.execute(InboundTask.java:44)
at org.wso2.carbon.mediation.ntask.NTaskAdapter.execute(NTaskAdapter.java:90)
at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not delete "c:\Test\ST\in\test.xml".
at org.apache.commons.vfs2.provider.local.LocalFile.doDelete(LocalFile.java:127)
at org.apache.commons.vfs2.provider.AbstractFileObject.deleteSelf(AbstractFileObject.java:838)
... 16 more
You can use Call mediator instead of send mediator. When the response is received, the mediation flow resumes from the next mediator in the sequence. So you can redirect or process the incoming response.

WSO2 ESB: HTTPS API

Is it possible to enable HTTPS protocol for an API on the WSO2 ESB?
https://localhost:8280/api/mySslApi
You can simply call the HTTPS endpoint of the API and it's enabled by default. Note that HTTPS port is 8243 not 8280.
https://localhost:8243/api/mySslApi
By default it is already enabled that way, if you follow axi2.xml config ,you will find below details,
1.for Http
<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
<parameter name="port" locked="false">8280</parameter>
<parameter name="non-blocking" locked="false">true</parameter>
<!--parameter name="bind-address" locked="false">hostname or IP address</parameter-->
<!--parameter name="WSDLEPRPrefix" locked="false">https://apachehost:port/somepath</parameter-->
<parameter name="httpGetProcessor" locked="false">org.wso2.carbon.mediation.transport.handlers.PassThroughNHttpGetProcessor</parameter>
<!--<parameter name="priorityConfigFile" locked="false">location of priority configuration file</parameter>-->
</transportReceiver>
2.for Https
<transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">
<parameter name="port" locked="false">8243</parameter>
<parameter name="non-blocking" locked="false">true</parameter>
<parameter name="HttpsProtocols">TLSv1,TLSv1.1,TLSv1.2</parameter>
Yes, In WSO2 ESB, when we want to secure a REST API, we can simply make it available via HTTPS like bellow.
<resource uri-template="/view/{symbol}" methods="GET" protocol="https">
And let the security handlers validate the credentials.
<handlers>
<handler class="org.wso2.rest.BasicAuthHandler"/>
</handlers>
For detail information : https://docs.wso2.com/display/ESB490/Securing+APIs
We currently support BasicAuth/usernametoken secured endpoints via the publisher..You may need to edit the axis2.xml for "HostnameVerifier" parameter in caseof handshake errors..Check the following link for further details.
http://docs.wso2.org/wiki/display/AM131/Adding+an+API