Java jsch and resuming file upload after interruption? - jsch

Question

I'm currently using in my code the com.jcraft.jsch library so that I can upload a file(or multiple files) from my local machine to a certain remote one. With file sizes of 5KB, 100KB, 200KB I don't have any concerns. However, I have one big concern when I tend to upload a file with file size 500MB , 1GB, 2GB and above, because there is always the possibility that the internet connection could fail on either side (local machine or remote)
I did a little research of my own and found that the Library has a field called RESUME, which refers to "file transfer mode" , but I haven't found an explanation about its proper use.
So my question is : Is there a way if the connection fails , after it is fixed, the file transfer continues from the point it was interrupted ?

Answer 1

I just "solved" this on my application and thought I would share what I learned.
I looked into how the RESUME is working and found that it depends on which methods you are using. For me I was using both a PipedInputStream and a PipedOutputStream since I might be transferring to/from local files or even both to/from remote servers.
I found that for me I provided my PipedOutputStream to the get method with no mode provided (defaults to OVERWRITE) and then provided my PipedInputStream to the put method with a parameter of RESUME. The put method progressed my InputStream the number of bytes equal to the current size of the file I am sending to.
This took a while as I was already progressing my PipedOutputStream X number of bytes and then the PipedInputStream was progressing another X bytes and I was getting significant gaps. I found this out by looking at the ChannelSftp source code
Of course this will be different if you are not doing the exact same thing as me, but if your source or destination are local you may not need to worry about that. I would try looking at the source code if you can't figure out how you are doing it.
I am using Grails so this may not work exactly for you, but here is what I did
/*
* This was initially copied from
* <a href="http://www.intelligrape.com/blog/2013/04/04/using-ftp-with-grails/">
* http://www.intelligrape.com/blog/2013/04/04/using-ftp-with-grails/</a> for
* the basic structure. JavaDoc and additional method were added as needed.
*
* #author Puneet Behl
* #author jonathan.tinsman
*/
class FtpService {
/**
* Gets the file from the server and loads it into the provided output stream
*
* #param outputStream
* - the output stream to have the file loaded to
* #param fileName
* - the desired file
* #param ftpCredential
* -the server credentials
*/
def load(OutputStream outputStream, String fileName, FtpCredential ftpCredential) {
connect(ftpCredential) { ChannelSftp sftp ->
sftp.get fileName, outputStream, new FtpMonitor()
}
}
/**
* Writes the file on the server
*
* #param inputStream
* - the input stream for writing
* #param fileName
* - the file name
* #param mode
* - the mode for the transfer (defaults to {#link ChannelSftp#OVERWRITE}
* #param ftpCredential
* - the server credentials
*/
def save(InputStream inputStream, String fileName, Integer mode = ChannelSftp.OVERWRITE, FtpCredential ftpCredential) {
connect(ftpCredential) { ChannelSftp sftp ->
sftp.put inputStream, fileName, mode
}
}
/**
* Transfers the file from the input server to the output server.
* <p>
* The usage of {#link PipedInputStream} and {#link PipedOutputStream} is
* from OsterMiller.org
*
* #param fileName
* - the file name
* #param inputFtpCredential
* - the input server
* #param outputFtpCredential
* - the output server
* #param mode
* - the mode for the transfer (defaults to {#link ChannelSftp#OVERWRITE}
*/
def transfer(String fileName, FtpCredential inputFtpCredential, FtpCredential outputFtpCredential, Integer mode = ChannelSftp.OVERWRITE) {
// To change the size of the buffer, add an int with the desired pipe
// size. The default is 1024
PipedInputStream input = new PipedInputStream();
PipedOutputStream output = new PipedOutputStream(input);
// Starting in different threads so they do not deadlock each other
new Thread(
new Runnable(){
public void run(){
new FtpService().load output, fileName, inputFtpCredential
}
}
).start();
/*
* only passing the mode to the "save" as the save will progress the
* input stream on it's own.
*
* If we pass the mode to the "load" method, then there will be a gap
* in the data as the "load" will progress the stream xx bytes and the
* "save" will progress it another xx bytes (the size of the existing
* file).
*/
save input, fileName, mode, outputFtpCredential
}
/**
* Connect to the server and call the provided ChannelSftp Closure.
*
* #param ftpCredential
* - the server to connect to
* #param closure
* - the closure to call
* #param disconnectOnFinish
* - to disconnect the Session when the Closure is done (defaults to true)
*/
private def connect(FtpCredential ftpCredential, Closure closure, boolean disconnectOnFinish = true) {
Session session = null
ChannelSftp sftp = null
try {
JSch jSch = new JSch()
session = jSch.getSession ftpCredential?.username, ftpCredential?.server, ftpCredential?.port
session.setConfig "StrictHostKeyChecking", "no"
if (ftpCredential?.password) {
session.password = ftpCredential?.password
} else {
File keyFile = new File("${grailsApplication.config.pathToKeyFile}")
jSch.addIdentity(keyFile?.absolutePath)
}
session.connect()
Channel sFtpChannel = session.openChannel "sftp"
sFtpChannel.connect()
sftp = sFtpChannel as ChannelSftp
sftp.cd ftpCredential?.remoteBaseDir
closure.call sftp
} catch (Exception ex) {
ex.printStackTrace()
} finally {
if (disconnectOnFinish) {
sftp?.exit()
session?.disconnect()
}
}
}
}