I was trying to check the size of a file before I download it using renci.sshnet in vb.net. The only thing I notice is the .length always return a value of 4096 but the actual file is 700mb. Could someone help me get the accurate file size?
Dim sftpfile = sftp.ListDirectory("/Data/fdb")
For Each ftpFile In sftpfile
If FileSize > (i want to check those higher than 50mb) Then
'output file size
End If
Next
You need to get a reference to the file as an SftpFile object. Then you can access the size attribute. Here's how in C# (converting to VB.NET shouldn't be hard):
SftpFile file = sftpClient.Get(filePath);
return file.Attributes.Size;
What you're probably reading is the internal buffer size (4096) which you shouldn't worry about.
Related
I'm trying to generate excel file from Oracle by FILE_UTIL. In Oracle document they gave some mode of operations like
W - Write
R - Read
WB -Write Byte
RB - Read Byte
Unable to understand the difference between W and WB. Thanks in advance.
The documentation you're referring to seems to the this, which says slightly more than you indicated in the question:
Specifies how the file is opened. Modes include:
r -- read text
w -- write text
a -- append text
rb -- read byte mode
wb -- write byte mode
ab -- append byte mode
The documentation also says:
byte_mode Indicates whether the file was open as a binary file, or as a text file
So the b indicates byte mode rather than text mode. The file is accessed as a character stream if it's in text mode, so the file should be encoded in the database character set, as it says in the operational notes for that package. And it's accessed as a binary stream in byte mode. Several methods, such as get_line, will raise an exception for a file opened in byte mode as a 'line' has no meaning for binary data.
So it you're processing a file that is text, which could be stored as a CLOB, then use the text-mode flags. If you're processing a file that contains binary data like an image or PDF, which could be stored as a BLOB, use the byte-mode flags.
Excel files contain binary data whether you have a .xls or .xlsx file, so you'd need to use byte mode. If you were generating a .csv file though, you'd probably want text-mode.
I have 500,000 files which I need to read in Fortran and each file has ~14,000 entries in it (each entry is only about 100 characters long). I need to process each line for each file at a time. For example, I need to process line 1 for all 500,000 files before moving on to line 2 from the files and so forth.
I cannot open them all at once (I tried making an array of file pointers and opening them all) because there will be too many files open at once. Instead, I would like to do something as follows:
do iline = 1,Nlines
do ifile = 1,Nfiles
! open the file
! read a line
! close the file
enddo
end
In hopes that this would allow me to read one line at a time (from each file) and then move on to the next line (in each file). Unfortunately, each time I open the file it starts me off at line 1 again. Is there any way to open/close a file and then open it again where you left off previously?
Thanks
Unfortunately it is not possible in this way in standard Fortran. Even If you specify
position="ASIS"
the actual position will be unspecified for a not already connected unit and will be in fact the beginning of the file on most systems.
That means You have to use
read(*,*)
enough times to get on the right place in the file.
You could also use stream access. The file would be again opened at the beginning, but you can use
read(u,*,pos=n) number
where n is the position saved from the previous open. You can get the position from
inquire(unit=u, pos=n)
n = n
You would open the file with acess="STREAM".
Also 500000 opened files is indeed too much. There are ways how to inquire for the system limits and how to control them, but also your compiler may have some limits http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Other solution: Couldn't you store the content of the files in memory? Today couple of Gigabytes is OK, but it may be not enough for you.
You can try using fseek and ftell in something like the following.
! initialize an array of 0's
do iline = 1,Nlines
do ifile = 1,Nfiles
! open the file
! fseek(fd, array(ifile))
! read a line
! array(ifile)=ftell(fd)
! close the file
enddo
end
The (untested) idea is to store the offset of each file in an array and position the cursor at that place upon opening the file. Then, once a line is read, the ftell retrieves the current position which is saved to memory for next round. If all entries have the same length, you can spare the array and just store one value.
If the files have fixed, i.e., constant, record lengths, you could use direct access. Then you could "directly" read a specific record. A big "if" however.
the overhead of all the file opening/closing will be a big performance bottleneck.
You should try to read as much as you can for each open operation given whatever memory you have:
pseudocode:
loop until done:
loop over all files:
open
fseek !as in damiens answer
read N lines into array ! N=100 eg.
save ftell value for file
close
end file loop
loop over N output files:
open
write array data
close
I ran into this problem when uploading a file with a super long name - my database field was only set to 50 characters. Since then, I have increased my database field length, but I'd like to have a way to check the length of the filename before uploading. Below is my code. The validation returns '85' as the character length. And it returns the same count for every different file I upload (none of which have a file name length of 85).
<cfscript>
missing_info = "<p>There was a slight problem with your submission. The following are required or invalid:</p><ul>";
// Check the length of the file name for our database field
if ( len(Form["ResumeFile1"]) gt 100 )
{
missing_info = missing_info & "<li>'Resume File 1' is invalid. Character length must be less than 100. Current count is " & len(Form["ResumeFile1"]) & ".</li>";
validation_error = true;
ResumeFileInvalidMarker = true;
}
</cfscript>
Anyone see anything wrong with this?
Thanks!
http://www.cfquickdocs.com/cf9/#cffile.upload
After you upload the file, the variable "clientFileName" will give you the name of the uploaded file, without a file extension.
The only way to read the filename before you upload it would be to use JavaScript to read and parse the value (file path) in the file field.
A quick clarification in the wording of your question. By the time your code executes the file upload has already happened. The file resides in a temporary directory on the ColdFusion server and the form field related to the file upload contains the temporary filename for that file. Aside from checking to see if a file has been specified, do not do anything directly with that file or you'll be circumventing some built in security.
You want to use the cffile tag with the upload action (or equivalent udf) to move the temp file into a folder of your choosing. At that point you get access to a structure containing lots of information. Usually I "upload" into a temporary directory for the application, which should be outside of the webroot for security.
At this point you'll then want to do any validation against the file, such as filename length, file type, file size, etc and delete the file if it fails any checks. If it passes all checks then you move it into it's final destination which may be inside the webroot.
In your case you'll want to check the cffile structure element clientFile which is the original filename including extension (which you'll need to check, since an extension doesn't need to be present and can be any length).
Say I have a 5 GB file. I want to split it in the following way.
First 100 MB is on the file
The rest go some reserve file
I do not want to use readalllines kind of function because it's too slow for large files.
I do not want to read the whole file to the memory. I want the program to handle only a medium chunk of data at a time.
You may use BinaryReader class and its method to read file in chunks.
Dim chunk() As Byte
chunk = br.ReadBytes(1024)
I'm using this method by Dave DeLong to calculate the size of a folder using the Carbon File Manager API:
http://github.com/davedelong/BuildCleaner/blob/b2712242b4eea1fff0e78a08b393a417e3019c8a/NSFileManager+FileSize.m
(Its the first method there)
The issue I'm having is that some folders (.app bundles in my case) are not reporting the correct size. For example, DiskWarrior is 8.2MB, and the method reports 6.6MB
Any ideas on why this is happening?
THANKS
I've improved on that source code of mine that you've linked to. Here's the newer version:
http://github.com/davedelong/BetterInfo/blob/aa1cfe079dad6207a7ddac84b108a768c2cc7156/NSFileManager+BetterInfo.m (You'll also need the corresponding .h file and this support file)
Now instead of returning and NSUInteger, it returns a struct of type "BIItemSyze", which has six members: dataLogicalSize, dataPhysicalSize, resourceLogicalSize, resourcePhysicalSize, logicalSize, and physicalSize.
The Finder will report file sizes rounded to the nearest multiple of the block size (usually 4 KB) followed by the real size in bytes, and many (most) applications are bundles of files, so the true size of the application may be far smaller than the size shown as the first ("on disk") value.
You can test this out by doing something (in the Terminal) like:
echo -n 'foo' > foo.txt
If you Get Info on this file in the Finder, it will report the size as "4 KB on disk (3 bytes)".
If you know how to use applescript's in your code, here's a method o return the size that you'd see in the Finder Get Info window. Note that the returned value is in bytes.
on getSizeInBytesFromPosixPath(posixPath)
try
set b to (POSIX file posixPath) as string
tell application "Finder" to return size of (b as alias)
on error
return 0
end try
end getSizeInBytesFromPosixPath