I am trying to upload file from local disk to remote server by com.jcraft.jsch.ChannelSftp.put(String src, String dst, SftpProgressMonitor monitor)
I find that even the upload progress is not completed,if there is no exception occured,program will go ahead.For example,send a message to client that a file already uploaded to server.But the file is still being uploaded,if client try to get the file immediately,the file is a blank file or inconsistent file.
How to pause the main program while uploading file,and resume it once complete upload?
JSch's ChannelSftp does the actual file transfer in a background thread, if you use that put method. This is done so your program can do other things during that, or even use the channel to upload or download other stuff.
The progress monitor object you did pass will be informed of the progress, and the end of the upload or download.
Use this to know when it is done, or when you can do other things with the file.
Related
I have a very basic question about file upload with embedded Jetty.
When I upload a large file, I know that Jetty server buffers it somewhere. Where does this buffer exist?
Also, Is there a way to disable the buffer and stream the request data directly to a destination such as HDFS?
If we can't disable this buffering, at least I need to have a control on deleting the tmp file after the file upload is complete.
Thanks in advance for the help.
That's defined by the servlet spec location value found in the MultipartConfigElement, typically declared as an #MultipartConfig annotation on the servlet you have receiving the uploaded files.
I have 2 machines in our datacenter:
The public server exposes part of the internal servers's storage through ftp. When files are uploaded to the ftp, the files in fact end up on the internal storage. But when watching the inotify events on the internal server's storage, i notice the file gets written in chunks, probably due to buffering at client side. The software on the internal server, watches the inotify events, to determine if new files have arrived. But due to the NFS manner of writing the files, there is no good way of telling when a file is complete. Is there a way of telling the NFS client to write files in only one operation, or is there a work around for this behaviour?
EDIT:
The events i get on the internal server, when uploading a file of around 900 MB are:
./ CREATE big_buck_bunny_1080p_surround.avi
# after the CREATE i get around 250K MODIFY and CLOSE_WRITE,CLOSE events:
./ MODIFY big_buck_bunny_1080p_surround.avi
./ CLOSE_WRITE,CLOSE big_buck_bunny_1080p_surround.avi
# when the upload finishes i get a CLOSE_NOWRITE,CLOSE
./ CLOSE_NOWRITE,CLOSE big_buck_bunny_1080p_surround.avi
of course, i could listen to the CLOSE_NOWRITE event, but reading inotify documentation says:
close_nowrite
A watched file or a file within a watched directory was closed, after being opened in read-only mode.
Which is not exactly the same as 'the file is complete'. The only workaround I see, is to use .part or .filepart files and move them, once uploaded, to the original filename and ignore the .part files in my storage watcher. Disadvantage is I'll have to explain this to customers, how to upload with .part. Not many ftp clients support this by default.
Basically, if you want to check when the write operations is completed, monitor the event IN_CLOSE_WRITE.
IN_CLOSE_WRITE gets "fired" when a file gets closed which was open for writing. Even if the file gets transferred in chunks, the FTP server will close the file only after the whole file has been transferred.
My application is keeping watch on a set of folders where users can upload files. When a file upload is finished I have to apply a treatment, but I don't know how to detect that a file has not finish to upload.
Any way to detect if a file is not released yet by the FTP server?
There's no generic solution to this problem.
Some FTP servers lock the file being uploaded, preventing you from accessing it, while the file is still being uploaded. For example IIS FTP server does that. Most other FTP servers do not. See my answer at Prevent file from being accessed as it's being uploaded.
There are some common workarounds to the problem (originally posted in SFTP file lock mechanism, but relevant for the FTP too):
You can have the client upload a "done" file once the upload finishes. Make your automated system wait for the "done" file to appear.
You can have a dedicated "upload" folder and have the client (atomically) move the uploaded file to a "done" folder. Make your automated system look to the "done" folder only.
Have a file naming convention for files being uploaded (".filepart") and have the client (atomically) rename the file after upload to its final name. Make your automated system ignore the ".filepart" files.
See (my) article Locking files while uploading / Upload to temporary file name for an example of implementing this approach.
Also, some FTP servers have this functionality built-in. For example ProFTPD with its HiddenStores directive.
A gross hack is to periodically check for file attributes (size and time) and consider the upload finished, if the attributes have not changed for some time interval.
You can also make use of the fact that some file formats have clear end-of-the-file marker (like XML or ZIP). So you know, that the file is incomplete.
Some FTP servers allow you to configure a hook to be called, when an upload is finished. You can make use of that. For example ProFTPD has a mod_exec module (see the ExecOnCommand directive).
I use ftputil to implement this work-around:
connect to ftp server
list all files of the directory
call stat() on each file
wait N seconds
For each file: call stat() again. If result is different, then skip this file, since it was modified during the last seconds.
If stat() result is not different, then download the file.
This whole ftp-fetching is old and obsolete technology. I hope that the customer will use a modern http API the next time :-)
If you are reading files of particular extensions, then use WINSCP for File Transfer. It will create a temporary file with extension .filepart and it will turn to the actual file extension once it fully transfer the file.
I hope, it will help someone.
This is a classic problem with FTP transfers. The only mostly reliable method I've found is to send a file, then send a second short "marker" file just to tell the recipient the transfer of the first is complete. You can use a file naming convention and just check for existence of the second file.
You might get fancy and make the content of the second file a checksum of the first file. Then you could verify the first file. (You don't have the problem with the second file because you just wait until file size = checksum size).
And of course this only works if you can get the sender to send a second file.
I have an issue where occasionally I need to work at Starbucks.
When I upload a PHP file the connection is slow so if a user tries to access the PHP file while I am uploading it they will of course be issues a fatal error.
This is very inconvenient to my busy websites. Is there a way that when a file is uploaded it can be uploaded to a temporary location, and then the server moves it to the real location once finished?
You can make WinSCP upload the file to temporary file and rename it once transfer completes automatically.
In Preferences go to the Transfer > Endurance tab and select All Files in the Enable ... Transfer to temporary file name box.
For details refer to:
https://winscp.net/eng/docs/ui_pref_resume
Why don't you just upload the file to a temporary folder on the server and execute commands on the server to remove the old file and move the new file? It should move the file fast enough on the server to eliminate any hiccups the users would see unless their timing was just right.
I've a client-server application on Mac. Client uploads a file and server downloads the file.
Sever reads a specific size of bytes from the client and write into the file. But in the middle user can delete the file using Finder context menu or from terminal. I want to stop any write/execution operation on this file from any other application till the download runs. It can be easily done using FILE_SHARE_READ while creating the file on Windows. But how we can achieve the same functionality on Mac?
I've tried advisory locks on Mac but no luck. If process A uses advisory locks on the file then process B can't access it but i can always delete the file using Finder context menu.
Are you sure you need to do this? So long as you've got an open file handle, it doesn't matter if the file is deleted, you can still read from it until you close the file handle. If the user deletes the file mid-transmission, it won't stop you from sending the full file.