Is there a way to configure a File connector for use in cloudhub, specifically related to reading in a file over FTPS and putting it into a file before beginning the actual processing of the contents?
Clarification:
I'm in cloudhub, which does not provide a filesystem in the same sense that a local/on-prem Mule setup has. One standard practice when dealing with streams (FTPS or similar) in order to avoid processing over the open stream is to take the incoming stream and use the File connector (outbound in this case) to put the inbound stream into a file, and then use that file for your flow process. How is this managed in CloudHub?
File Connector is to read files from paths specified on the server. They cannot be used to read from remote servers.
I case you want to have a File to start your flow with try the following.
<flow name="ftp_reader_flow">
<ftp: inbound> Read from the remote directory
...
<file:outbound> to a local directory
</flow>
<flow name="actual_processing_flow">
<file:inbound> read from the local directory.
... Continue with the processing
.....
</flow>
Hope this helps.
You can use the connector for temporary data with the tmp directory.
From the MuleSoft Documentation:
Disk Persistence
CloudHub does not guarantee that writing to disk survives hardware
failures. Instead, you must use an external storage mechanism to store
information. For small amounts of data, you can use the Object Store.
For applications that have large data storage requirements, we
recommend use of a cloud service such as Amazon S3. For temporary
storage, the File connector is still available and can be used with
the /tmp directory.
You can use File Connector in CloudHub as well, But Make sure your are reading or writing the file from classpath -src/main/resource or any folder from project classpath only.
Related
I want to store data on cloudhub in a temporary file. Is it possible to write and read from this /tmp directory? if yes, can someone tell me how ?
The recommended way is to use Object store https://docs.mulesoft.com/object-store/
The /tmp directory may get lost if there is any outage or redeployment or anything which kills the worker. To access it you can use File connector read/write.
It's not a good idea to use /tmp directory to store application-specific production data (that too 53,000 records!!). That /tmp directory is meant for the app to store files and objects temporarily created during app's run, such as temp files created by dataweave during processing of large files.
The better option to deal with such a scenario is to use ObejctStore v2 which supports maxEntries, expirationInterval and entryTTL when used in CloudHub. You can maintain unlimited number of key/value pairs per application which the size of each value can be up to 10MB. See here for more details on ObjStore v2.
I need to copy the directory (/tmp/xxx_files/xxx/Output) head containing sub folders and files from HDFS (Hadoop distributed file system). I'm using HDFS connector but it seems it does not support this.
It always getting an error like:
org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Path is not a file: /tmp/xxx_files/xxx/Output/
I don't see any option is HDFS connector for copying the files/directories inside the path specified. It is always expecting file names to be copied.
Is it possible to copy a directory head containing sub-folders and files using the HDFS connector from MuleSoft?
As the technical documentation of the HSFS connector on the official MuleSoft website states, the code is hosted at the GitHub site of the connector:
The Anypoint Connector for the Hadoop Distributed File System (HDFS)
is used as a bi-directional gateway between applications. Its source
is stored at the HDFS Connector GitHub site.
What it does not state, that there is also a more detailed technical documentation available on the GitHub site.
Here you can also find different examples how to use the connector for basic file-system operations.
The links seem to be broken in the official MuleSoft documentation.
You can find the repository here:
https://github.com/mulesoft/mule-hadoop-connector
The operations are implemented in the HdfsOperations java class. (See also the FileSystemApiService class)
As you can see, the functionality you expect is not implemented. It is not supported out-of-the-box.
You can't copy a directory head containing sub folders and files from HDFS without any further effort using the HDFS connector.
I want to read some files from samba server, it looks like smb protocol is not one of the supported protocols by mule components like ftp, sftp etc.. So can anyone share some ideas including example implementation for doing this? Let us assume we want to read a csv file and save it to payload so that it can be used to pass to weave transform component.
Typically, you would mount the share via the operating system and then start the Mule Runtime process in the context of a user/grp that has the proper permissions to rw.
I am new to Mule, so let me say if this is a very basic question then please excuse me.
I am trying to copy a file (size: > 1GB) from one FTP Server(Source) to another FTP Server(Destination).
It takes 1 min to copy the file into the location at which my mule is polling.
As soon as i start copying the file on that location then itself my mule file copy is triggered and tries to read that file and fails as it is still is use by another process.
I wish to make a system in which the file copy is triggered not only when the the mule can detect a file at the Source, but i also want mule to wait for the file to be ready for reading and then start copying it. I don't want to insert a delay of fixed amount for this prupose.
Someone please suggest a method of doing it.
The problem with that readiness is that there is no such kind of standardised flag in the filesystem. So the easier way of doing it is using the ´fileAge´ attribute of the ftp connector, that is sadly EE only.
Can't you move the file to the ftp directory Mule is pollong once is ready to be processed? That wouldn't require copying the file again, so it takes almost no time. Mule would pick it up only when is ready to be processed.
It seems cloudbees writes the logs only to stream and not to a file. I need to save my logs. Can I use any option other than papertrail to store/retreive log files? Can I listen to the some input stream and get feed of logs? Can I dump logs directly to Amazon S3?
As filesystem isn't persistent we also don't provide file based logging. We don't provide a platform helper to store logs to S3, as papertrails offers a comparable persistent solution with better performances and dedicated service.
You can for sure use your favorite logging framework and custom extensions to get log stored on S3 or other if you prefer this option.