I have the following code:
With context.Response
Dim req As HttpWebRequest = WebRequest.Create("http://www.Google.com/")
req.Proxy = Nothing
Dim res As HttpWebResponse = req.GetResponse()
Dim Stream As Stream = res.GetResponseStream
.OutputStream.Write(Stream, 0, Stream.Length)
End With
Sadly, the above code doesn't work. I need to take the RequestStream and put it into the OutputStream from the context.Response.
Any ideas?
Write takes a byte array, whereas you are passing it a stream.
Try reading from the stream and getting all the data before writing it back.
First, read the data into an intermediate byte array (Taken from here):
Dim bytes(Stream.Length) As Byte
Dim numBytesToRead As Integer = s.Length
Dim numBytesRead As Integer = 0
Dim n As Integer
While numBytesToRead > 0
' Read may return anything from 0 to 10.
n = Stream.Read(bytes, numBytesRead, 10)
' The end of the file is reached.
If n = 0 Then
Exit While
End If
numBytesRead += n
numBytesToRead -= n
End While
Stream.Close()
Then write it to the output stream:
.OutputStream.Write(bytes, 0, Stream.Length)
Related
I am reading files using stream reader and writer with a buffer. Is it possible to calculate the crc32 checksum of the file at the same time so that it would evaluate the buffer during each iteration, and the result would be the crc32 of the complete file, instead of having to read teh file twice? Here is the code:
Public Function APPEND_FILE_TO_FILE(ByVal SOURCE_FILE As String, ByVal DESTINATION_FILE As String)
Dim nn As New FileInfo(SOURCE_FILE)
Dim BUFFER(BUFFER_SIZE) As Byte
Dim BYTES_READ As Long = 0
Using inFile As New System.IO.FileStream(SOURCE_FILE, IO.FileMode.Open, IO.FileAccess.Read)
Using outFile As New System.IO.FileStream(DESTINATION_FILE, IO.FileMode.OpenOrCreate, IO.FileAccess.Write)
Do
BYTES_READ = inFile.Read(BUFFER, 0, BUFFER_SIZE)
TOTAL_PROCESSED_DATA += BYTES_READ ' GUI related
outFile.Write(BUFFER, 0, BYTES_READ)
Loop While BYTES_READ > 0
End Using
End Using
End Function
CRC32 function:
Public Function GetCRC32(ByVal sFileName As String) As String
Try
Dim FS As FileStream = New FileStream(sFileName, FileMode.Open, FileAccess.Read, FileShare.Read, 8192)
Dim CRC32Result As Integer = &HFFFFFFFF
Dim Buffer(4096) As Byte
Dim ReadSize As Integer = 4096
Dim Count As Integer = FS.Read(Buffer, 0, ReadSize)
Dim CRC32Table(256) As Integer
Dim DWPolynomial As Integer = &HEDB88320
Dim DWCRC As Integer
Dim i As Integer, j As Integer, n As Integer
'Create CRC32 Table
For i = 0 To 255
DWCRC = i
For j = 8 To 1 Step -1
If (DWCRC And 1) Then
DWCRC = ((DWCRC And &HFFFFFFFE) \ 2&) And &H7FFFFFFF
DWCRC = DWCRC Xor DWPolynomial
Else
DWCRC = ((DWCRC And &HFFFFFFFE) \ 2&) And &H7FFFFFFF
End If
Next j
CRC32Table(i) = DWCRC
Next i
'Calcualting CRC32 Hash
Do While (Count > 0)
For i = 0 To Count - 1
n = (CRC32Result And &HFF) Xor Buffer(i)
CRC32Result = ((CRC32Result And &HFFFFFF00) \ &H100) And &HFFFFFF
CRC32Result = CRC32Result Xor CRC32Table(n)
Next i
Count = FS.Read(Buffer, 0, ReadSize)
Loop
Return Hex(Not (CRC32Result))
Catch ex As Exception
Return ""
End Try
End Function
The above code leaves the file being parsed [open] and cannot be renamed or deleted.
The line:
Count = FS.Read(Buffer, 0, ReadSize)
Loop
Return Hex(Not (CRC32Result))
Should Be:
Count = FS.Read(Buffer, 0, ReadSize)
Loop
FS.Close
Return Hex(Not (CRC32Result))
Then the file can be manipulated again.
I am wanting some help with some code please. Here is my code:
Public Function ReadBinaryFileLarge(strFilename As String) As Byte()
Dim position As Integer = 0
Dim bufferSize As Integer = 4096
Dim bytes() As Byte
Using fsOpen As FileStream = New FileStream(strFilename, FileMode.Open)
ReDim bytes((fsOpen.Length) - 1)
Do
If (position + bufferSize) > fsOpen.Length Then
fsOpen.Read(bytes, position, fsOpen.Length - position)
Exit Do
Else
fsOpen.Read(bytes, position, bufferSize)
End If
position += bufferSize
Application.DoEvents()
Loop
End Using
Return bytes
End Function
Public Sub SaveBinaryFileLarge(strFilename As String, bytesToWrite() As Byte)
Dim position As Integer = 0
Using fsNew As FileStream = New FileStream(strFilename, FileMode.Create, FileAccess.Write)
Do
Dim intToCopy As Integer = Math.Min(4096, bytesToWrite.Length - position)
Dim buffer(intToCopy - 1) As Byte
Array.Copy(bytesToWrite, position, buffer, 0, intToCopy)
fsNew.Write(buffer, 0, buffer.Length)
position += intToCopy
Application.DoEvents()
Loop While position < bytesToWrite.Length
End Using
End Sub
My problem is that with large files, the byte array cannot be declared that large. I am needing to load it into a byte array to do some encryption work on the byte array. I am playing around with this code:
Public Function ReadBinaryFileTest(strFilename As String) As Byte()()
Const SIZE As Integer = &H1000 '4096 <-experiment with this value
Dim bytes()() As Byte
Using fsOpen As FileStream = New FileStream(strFilename, FileMode.Open, FileAccess.Read, FileShare.Read, SIZE)
Dim ubound = (fsOpen.Length / SIZE) - 1
bytes = new byte(ubound)()
Dim read As Integer = 0
For index As Integer = 0 To ubound
bytes(index) = New Byte(SIZE - 1)
read = fsOpen.Read(bytes(index), 0, SIZE)
If read <> SIZE Then
Array.Resize(bytes(index), read) 'this should only happen once if at all
End If
Next
End Using
Return bytes
End Function
However I am getting these errors:
bytes = new byte(ubound)()
'{' expected
and
bytes(index) = New Byte(SIZE - 1)
Type 'Byte' has no constructors.
Is this the correct way to go about this problem? If not, how should I attack it?
You have to use ReDim,
ReDim bytes(ubound-1)
Or
bytes = New Byte(ubound)() {}
I have this code which works well:
Public Function LoadBinaryFile(strFilename As String) As Byte()
Using fsSource As FileStream = New FileStream(strFilename, FileMode.Open, FileAccess.Read)
' Read the source file into a byte array.
Dim bytes() As Byte = New Byte((fsSource.Length) - 1) {}
Dim numBytesToRead As Integer = CType(fsSource.Length, Integer)
Dim numBytesRead As Integer = 0
'tsProgressBar.Minimum = 0
'tsProgressBar.Maximum = numBytesToRead
While (numBytesToRead > 0)
' Read may return anything from 0 to numBytesToRead.
Dim n As Integer = fsSource.Read(bytes, numBytesRead, _
numBytesToRead)
' Break when the end of the file is reached.
If (n = 0) Then
Exit While
End If
numBytesRead = (numBytesRead + n)
numBytesToRead = (numBytesToRead - n)
'tsProgressBar.Value = numBytesRead
End While
numBytesToRead = bytes.Length
Return bytes
End Using
End Function
And I have this code to save the file which also works well:
Public Function SaveBinaryFile(strFilename As String, bytesToWrite() As Byte) As Boolean
Using fsNew As FileStream = New FileStream(strFilename, FileMode.Create, FileAccess.Write)
fsNew.Write(bytesToWrite, 0, bytesToWrite.Length)
End Using
End Function
What I am after is some help to modify the SaveBinaryFile function to implement a progress bar.
Final:
OK, I have written the function myself. Here it is:
Public Function ReadBinaryFile(strFilename As String) As Byte()
Dim position As Integer = 0
Dim bufferSize As Integer = 4096
Dim bytes() As Byte
'frmMain.tsProgressBar.Value = 0
Using fsOpen As FileStream = New FileStream(strFilename, FileMode.Open)
redim bytes((fsOpen.Length) - 1)
Do
If (position + bufferSize) > fsOpen.Length Then
fsOpen.Read(bytes, position, fsOpen.Length - position)
Exit Do
Else
fsOpen.Read(bytes, position, bufferSize)
End If
'frmMain.tsProgressBar.Value = ((position / fsOpen.Length) * 100)
'frmMain.tsProgressBar.Refresh()
Application.DoEvents()
position += bufferSize
Loop
End Using
Return bytes
End Function
My.Computer.Filesystem.ReadAllBytes("filename") reads the entire file into a byte array.
My.Computer.Filesystem.WriteAllBytes("filename", bytes, false) writes it back.
I've never tried using the Async options on FileStream but there don't seem to be any event handlers.
I would break it down into a loop, you have the total amount of bytes to write so you could loop through writing 4k at a time, update your progress bar and continue the loop.
Public Sub SaveBinaryFile(strFilename As String, bytesToWrite() As Byte)
Dim position As Integer = 0
Using fsNew As FileStream = New FileStream(strFilename, FileMode.Create, FileAccess.Write)
Do
Dim intToCopy As Integer = Math.Min(4096, bytesToWrite.Length - position)
Dim buffer(intToCopy - 1) As Byte
Array.Copy(bytesToWrite, position, buffer, 0, intToCopy)
fsNew.Write(buffer, 0, buffer.Length)
ProgressBar1.Value = ((position / bytesToWrite.Length) * 100)
Application.DoEvents()
position += intToCopy
Loop While position < bytesToWrite.Length
End Using
End Sub
'Or, if you are reading a file from a URL, here is a way to read the file into an array of bytes.
Dim WebRequest As Net.HttpWebRequest = Net.WebRequest.Create("http://mypage.abc.com/myfolder/MyFileName.xls")
Using WBinReader As BinaryReader = New BinaryReader(WRequest.GetResponse.GetResponseStream)
Dim file_buffer() As Byte
'10000000 is an arbitrary number, but File_Buffer will have no more
'elements than they number of bytes in the URL's file.
file_buffer = WBinReader.ReadBytes(10000000)
file_bytes =UBound(buffer)+1
End Using
Brian Jasmer
I'm having some trouble with this, despite finding examples. I think it may be an encoding problem, but I'm just not sure. I am trying to programitally download a file from a https server, that uses cookies (and hence I'm using httpwebrequest). I'm debug printing the capacity of the streams to check, but the output [raw] files look different. Have tried other encoding to no avail.
Code:
Sub downloadzip(strURL As String, strDestDir As String)
Dim request As HttpWebRequest
Dim response As HttpWebResponse
request = Net.HttpWebRequest.Create(strURL)
request.UserAgent = strUserAgent
request.Method = "GET"
request.CookieContainer = cookieJar
response = request.GetResponse()
If response.ContentType = "application/zip" Then
Debug.WriteLine("Is Zip")
Else
Debug.WriteLine("Is NOT Zip: is " + response.ContentType.ToString)
Exit Sub
End If
Dim intLen As Int64 = response.ContentLength
Debug.WriteLine("response length: " + intLen.ToString)
Using srStreamRemote As StreamReader = New StreamReader(response.GetResponseStream(), Encoding.Default)
'Using ms As New MemoryStream(intLen)
Dim fullfile As String = srStreamRemote.ReadToEnd
Dim memstream As MemoryStream = New MemoryStream(New UnicodeEncoding().GetBytes(fullfile))
'test write out to flie
Dim data As Byte() = memstream.ToArray()
Using filestrm As FileStream = New FileStream("c:\temp\debug.zip", FileMode.Create)
filestrm.Write(data, 0, data.Length)
End Using
Debug.WriteLine("Memstream capacity " + memstream.Capacity.ToString)
'Dim strData As String = srStreamRemote.ReadToEnd
memstream.Seek(0, 0)
Dim buffer As Byte() = New Byte(2048) {}
Using zip As New ZipInputStream(memstream)
Debug.WriteLine("zip stream cap " + zip.Length.ToString)
zip.Seek(0, 0)
Dim e As ZipEntry
Dim flag As Boolean = True
Do While flag ' daft, but won't assign e=zip... tries to evaluate
e = zip.GetNextEntry
If IsNothing(e) Then
flag = False
Exit Do
Else
e.UseUnicodeAsNecessary = True
End If
If Not e.IsDirectory Then
Debug.WriteLine("Writing out " + e.FileName)
' e.Extract(strDestDir)
Using output As FileStream = File.Open(Path.Combine(strDestDir, e.FileName), _
FileMode.Create, FileAccess.ReadWrite)
Dim n As Integer
Do While (n = zip.Read(buffer, 0, buffer.Length) > 0)
output.Write(buffer, 0, n)
Loop
End Using
End If
Loop
End Using
'End Using
End Using 'srStreamRemote.Close()
response.Close()
End Sub
So I get the right size file downloaded, but dotnetzip does not recognise it, and the files that get copied out are incomplete/invalid zips. I've spent most of today on this, and am ready to give up.
I think the answer will be to break down the problem, and perhaps change a couple aspects in the code.
For example, lets get rid of converting the response stream to a string:
Dim memStream As MemoryStream
Using rdr As System.IO.Stream = response.GetResponseStream
Dim count = Convert.ToInt32(response.ContentLength)
Dim buffer = New Byte(count) {}
Dim bytesRead As Integer
Do
bytesRead += rdr.Read(buffer, bytesRead, count - bytesRead)
Loop Until bytesRead = count
rdr.Close()
memStream = New MemoryStream(buffer)
End Using
Next, there's an easier way to output the contents of a memory stream to a file. Consider your code
Dim data As Byte() = memstream.ToArray()
Using filestrm As FileStream = New FileStream("c:\temp\debug.zip", FileMode.Create)
filestrm.Write(data, 0, data.Length)
End Using
can be replaced with
Using filestrm As FileStream = New FileStream("c:\temp\debug.zip", FileMode.Create)
memstream.WriteTo(filestrm)
End Using
That eliminates the need to transfer your memory stream into another byte array, and then push the byte array down the stream, when in fact the memory stream can transfer data directly to file (via the filestream) saving the middle-man buffer.
I'll admit I haven't worked with the Zip/compression libraries you're using, but with the above amendments you have removed unnecessary transfers between streams, byte arrays, strings, etc, and hopefully eliminated the encoding issues you were having.
Give that a try and let us know how you get on. Consider attempting to open the file that you saved ("C:\temp\debug.zip") to see if it is listed as corrupt. If not, then you know at least as far as that in the code, it is working ok.
I thought I'd post my full working solution to my own question, it combines the two excellent replies I've had, thank you guys.
Sub downloadzip(strURL As String, strDestDir As String)
Try
Dim request As HttpWebRequest
Dim response As HttpWebResponse
request = Net.HttpWebRequest.Create(strURL)
request.UserAgent = strUserAgent
request.Method = "GET"
request.CookieContainer = cookieJar
response = request.GetResponse()
If response.ContentType = "application/zip" Then
Debug.WriteLine("Is Zip")
Else
Debug.WriteLine("Is NOT Zip: is " + response.ContentType.ToString)
Exit Sub
End If
Dim intLen As Int32 = response.ContentLength
Debug.WriteLine("response length: " + intLen.ToString)
Dim memStream As MemoryStream
Using stmResponse As IO.Stream = response.GetResponseStream()
'Using ms As New MemoryStream(intLen)
Dim buffer = New Byte(intLen) {}
'Dim memstream As MemoryStream = New MemoryStream(buffer)
Dim bytesRead As Integer
Do
bytesRead += stmResponse.Read(buffer, bytesRead, intLen - bytesRead)
Loop Until bytesRead = intLen
memStream = New MemoryStream(buffer)
Dim res As Boolean = False
res = ZipExtracttoFile(memStream, strDestDir)
End Using 'srStreamRemote.Close()
response.Close()
Catch ex As Exception
'to do :)
End Try
End Sub
Function ZipExtracttoFile(strm As MemoryStream, strDestDir As String) As Boolean
Try
Using zip As ZipFile = ZipFile.Read(strm)
For Each e As ZipEntry In zip
e.Extract(strDestDir)
Next
End Using
Catch ex As Exception
Return False
End Try
Return True
End Function
You can download into a MemoryStream, then examine it:
Public Sub Download(url as String)
Dim req As HttpWebRequest = System.Net.WebRequest.Create(url)
req.Method = "GET"
Dim resp As HttpWebResponse = req.GetResponse()
If resp.ContentType = "application/zip" Then
Console.Error.Write("The result is a zip file.")
Dim length As Int64 = resp.ContentLength
If length = -1 Then
Console.Error.WriteLine("... length unspecified")
length = 16 * 1024
Else
Console.Error.WriteLine("... has length {0}", length)
End If
Dim ms As New MemoryStream
CopyStream(resp.GetResponseStream(), ms) '' **see note below!!!!
'' list contents of the zip file
ms.Seek(0,SeekOrigin.Begin)
Using zip As ZipFile = ZipFile.Read (ms)
Dim e As ZipEntry
Console.Error.WriteLine("Entries:")
Console.Error.WriteLine(" {0,22} {1,10} {2,12}", _
"Name", "compressed", "uncompressed")
Console.Error.WriteLine("----------------------------------------------------")
For Each e In zip
Console.Error.WriteLine(" {0,22} {1,10} {2,12}", _
e.FileName, _
e.CompressedSize, _
e.UncompressedSize)
Next
End Using
Else
Console.Error.WriteLine("The result is Not a zip file.")
CopyStream(resp.GetResponseStream(), Console.OpenStandardOutput)
End If
End Sub
Private Shared Sub CopyStream(input As Stream, output As Stream)
Dim buffer(32768 - 1) As Byte
Dim n As Int32
Do
n = input.Read(buffer, 0, buffer.Length)
If n = 0 Then Exit Do
output.Write(buffer, 0, n)
Loop
End Sub
EDIT
Just one note - I would not advise using this code (this approach) if the Zip file is very large. How large is "very large"? Well that depends, of course. The code I suggested above downloads the file into a memory stream, which of course means the entire contents of the zip file are held in memory. If it is a 28kb zip file, then there's no problem. But if it is a 2gb zip file, then you may have a big problem.
In that case you will want to stream it to a temporary file on disk, not to a MemoryStream. I'll leave that as an exercise for the reader.
The above will work for "reasonably sized" zip files, where "reasonable" depends on your machine configuration and application scenario.
Friends, I am able to get XML file by sing bytes, perhaps which is getting some problem. Can u suggest me alternate method to do the same thing to save XML file?
Try
Dim strUrl As String = "http://example.com"
Dim wr As HttpWebRequest = CType(WebRequest.Create(strUrl), HttpWebRequest)
Dim ws As HttpWebResponse = CType(wr.GetResponse(), HttpWebResponse)
ws.ContentType = "UTF-16"
Dim str As Stream = ws.GetResponseStream()
Dim inBuf(100000) As Byte
Dim bytesToRead As Integer = CInt(inBuf.Length)
Dim bytesRead As Integer = 0
While bytesToRead > 0
Dim n As Integer = str.Read(inBuf, bytesRead, bytesToRead)
If n = 0 Then
Exit While
End If
bytesRead += n
bytesToRead -= n
End While
Dim fstr As New FileStream("c:/GetXml.xml", FileMode.OpenOrCreate, FileAccess.Write)
fstr.Write(inBuf, 0, bytesRead)
str.Close()
fstr.Close()
Catch ex As WebException
Response.Write(ex.Message)
End Try
Why not just use the WebClient class and its DownloadFile method?? Seems a lot easier....
This is in C#, but you should have no trouble converting that to VB.NET:
WebClient wc = new WebClient();
wc.DownloadFile("http://xyz", #"C:\getxml.xml");
and you're done!
Marc
Consider using XMLTextReader. This example just loads the entire XML into a string, but obviously you could write it to a file instead:
Dim strUrl As String = "http://xyz.com"
Dim reader As XmlTextReader = New XmlTextReader(strUrl)
Dim output as String
Do While (reader.Read())
Select Case reader.NodeType
Case XmlNodeType.Element
Output = Output + "<" + reader.Name
If reader.HasAttributes Then
While reader.MoveToNextAttribute()
Output = Output + " {0}='{1}'", reader.Name, reader.Value)
End While
End If
Output = Output + ">"
Case XmlNodeType.Text
Output = Output + reader.Value
Case XmlNodeType.EndElement
Output = Output + "</" + reader.Name + ">"
End Select
Loop
What if the service is sending the request to our URL? How do I adjust this to read the http stream they send? Having such a hard time... (Should I do a separate thread? Sorry.)