I am using an httpWebRequest in my windows app to download files from a webserver (sURL), to a local folder (fileDestination) as such:
Public Function DownloadFile(ByVal sURL As String, _
ByVal fileDestination As String, _
ByVal WebRequestType As String) As Boolean
Dim URLReq As HttpWebRequest
Dim URLRes As HttpWebResponse
Dim FileStreamer As FileStream
Dim bBuffer(999) As Byte
Dim iBytesRead As Integer
Dim folderDestination As String
Dim sChunks As Stream
Try
FileStreamer = New FileStream(fileDestination, FileMode.Create)
URLReq = WebRequest.Create(sURL)
URLRes = URLReq.GetResponse 'Error occurs here!!
sChunks = URLReq.GetResponse.GetResponseStream
DownloadProgressBar.Value = 0
DownloadProgressBar.Maximum = URLRes.ContentLength
Do
iBytesRead = sChunks.Read(bBuffer, 0, 1000)
FileStreamer.Write(bBuffer, 0, iBytesRead)
Loop Until iBytesRead = 0
sChunks.Close()
FileStreamer.Close()
URLRes.Close()
Return True
Catch ex As Exception
Return False
End Try
End Function
This works fine for the first few files. But then it starts giving the following error on the URLReq.GetResponse line:
"operation has timed out"
Anyone know what could be causing this?
If you're targeting the 1.1 framework and are running multiple concurrent requests try setting System.Net.ServicePointManager.DefaultConnectionLimit
The timeout is set to 10000 milliseconds (or 10 seconds). Is that long enough for the roundtrip to the web server?
The MSDN Documentation says the default is 100 seconds (100000 ms) is there any reason you changed that default?
Related
I am having a program in VB, which searches for image in a website as follows:
www.website.com/1.jpg | www.website.com/2.jpg | www.website.com/3.jpg
in loop, and the program opens up only 3.jpg in browser, since 1 and 2 jpg does not exit in server. The program is up and running, but is very very slow, around 120 searches in a minute. However one of my colleagues designed the same program in Angular and that program is running very fast, around 500/600 searches in a minute.
He told me, what he did is generated 50 asynchronous calls to server, and then again 50, and then again 50 and went on like this. thus making his program very fast and ovbiously accurate.
I studied and learnt, that in Visual Basic too, we have async and wait calls to server request, but I cannot figure out how.
Here goes my existing code. Can anyone help me.
Private Sub Button2_Click(sender As Object, e As EventArgs) Handles Button2.Click
Dim MYURL as string
itemsF = 0
While Not itemsF = 50000
MYURL = "www.website.com/" & itemsF & ".jpg"
CheckPageExists(MYURL)
itemsF=itemsF+1
End While
End Sub
Private Function CheckPageExists(ByVal url As String) As Boolean
Dim request As Net.HttpWebRequest
Dim response As Net.HttpWebResponse
request = Net.WebRequest.Create(url)
request.Timeout = 5000
Try
response = request.GetResponse()
Catch ex As Exception
'IMAGE DOES NOT EXITS
Exit Function
End Try
Process.Start(url)
End Function
First, you have to make CheckPageExists an Async method:
Private Async Function CheckPageExists(ByVal url As String) As Task(Of Boolean)
Dim request As Net.HttpWebRequest = Net.WebRequest.Create(url)
request.Timeout = 5
Dim Result As Boolean
Try
Using response As HttpWebResponse = Await request.GetResponseAsync.ConfigureAwait(False)
Using responseReader As New IO.StreamReader(response.GetResponseStream)
Dim actualResponse As String = Await responseReader.ReadToEndAsync
Result = True
End Using
End Using
Catch ex As Exception
'IMAGE DOES NOT EXITS
Result = False
End Try
Console.WriteLine(url)
''Process.Start("chrome.exe", url)
Return Result
End Function
As you can see, instead of GetResponse we are using GetResponseAsync, which is Async itself. This method is very similar to what you were doing before, I
just added Return statements for clarity and a StreamReader to read the response of your website.
Once you've done that, you just need to change your Button2_Click method to call this other method, which incorporates all you were doing before:
Async Function MakeRequests() As Task
Dim tasks As List(Of Task(Of Boolean)) = New List(Of Task(Of Boolean))
Dim itemsF As Integer = 5
For i = 1 To itemsF
Dim MYURL As String = "http://www.touchegolfschool.com/images/" & i & ".jpg"
tasks.Add(CheckPageExists(MYURL))
Next
While tasks.Select(Function(x) x.Result).Count < tasks.Count
Thread.Sleep(100)
End While
End Function
The main change was adding tasks.Select(Function(x) x.Result).Count < tasks.Count; what you were seeing before was a request made but never returning because of the time it took to return; telling the main function to wait until all requests have a result makes the application wait long enough for the responses to come.
Here the main difference is the use of Tasks.
Check this for official documentation on asynchronous programming.
This is the modified code about uploadsession soo at small size file it working like a charm but when I try larger file like 5mb up. The following error keep showing up :
+$exception {"lookup_failed/closed/..."} System.Exception {Dropbox.Api.ApiException}
Private Async Sub UploadToolStripMenuItem2_Click(sender As Object, e As EventArgs) Handles UploadToolStripMenuItem2.Click
Dim C As New OpenFileDialog
C.Title = "Choose File"
C.Filter = "All Files (*.*)|*.*"
If C.ShowDialog = Windows.Forms.DialogResult.OK Then
Dim fileinfos = Path.GetFileName(C.FileName)
Dim filetempat = Path.GetFullPath(C.FileName)
Dim tempat As String = direktori.Text & "/" & fileinfos
Await Upload(filetempat, tempat)
End If
End Sub
Async Function Upload(localPath As String, remotePath As String) As Task
Const ChunkSize As Integer = 4096 * 1024
Using fileStream = File.Open(localPath, FileMode.Open)
If fileStream.Length <= ChunkSize Then
Await A.Files.UploadAsync(remotePath, body:=fileStream)
Else
Await Me.ChunkUpload(remotePath, fileStream, ChunkSize)
End If
End Using
End Function
Private Async Function ChunkUpload(path As [String], stream As FileStream, chunkSize As Integer) As Task
Dim numChunks As Integer = CInt(Math.Ceiling(CDbl(stream.Length) / chunkSize))
Dim buffer As Byte() = New Byte(chunkSize - 1) {}
Dim sessionId As String = Nothing
For idx As Integer = 0 To numChunks - 1
Dim byteRead = stream.Read(buffer, 0, chunkSize)
Using memStream = New MemoryStream(buffer, 0, byteRead)
If idx = 0 Then
Dim result = Await A.Files.UploadSessionStartAsync(True, memStream)
sessionId = result.SessionId
kondisi.Text=byteRead
Else
Dim cursor = New UploadSessionCursor(sessionId, CULng(CUInt(chunkSize) * CUInt(idx)))
If idx = numChunks - 1 Then
Dim fileMetadata As FileMetadata = Await A.Files.UploadSessionFinishAsync(cursor, New CommitInfo(path), memStream)
MessageBox.Show("Upload Complete")
Console.WriteLine(fileMetadata.PathDisplay)
Else
Await A.Files.UploadSessionAppendV2Async(cursor, True, memStream)
MessageBox.Show("Upload Failed")
End If
End If
End Using
Next
End Function
okay now its fixed, but when i got home, and try this code at my home,i got error, is this method are affected by slow internet connection ?.cause my campus have a decent speed.
this is the error message
+$exception {"Cannot access a disposed object.\r\nObject name: 'System.Net.Sockets.Socket'."} System.Exception {System.ObjectDisposedException}
this Cannot obtain value of local or argument '<this>' as it is not available at this instruction pointer, possibly because it has been optimized away. System.Net.Sockets.Socket
asyncResult Cannot obtain value of local or argument 'asyncResult' as it is not available at this instruction pointer, possibly because it has been optimized away. System.IAsyncResult
errorCode Cannot obtain value of local or argument 'errorCode' as it is not available at this instruction pointer, possibly because it has been optimized away. System.Net.Sockets.SocketError
this error window i got
A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
Additional information: Cannot access a disposed object.
This Closed error indicates you can't continue uploading to the upload session because it's already been closed.
The UploadSessionStartAsync and UploadSessionAppendV2Async both take a bool parameter called close, which closes the session.
You're always setting that to True when you call those, closing the sessions. You shouldn't close a session until you're finished uploading data for it.
I am trying to upload a file using a thread. I have placed a simple file upload control and a button on a page.The code looks like this-
Protected Sub btnUpload_Click(ByVal sender As Object,
ByVal e As EventArgs) Handles btnUpload.Click
Dim timeStart As TimeSpan = Nothing
Dim timeEnd As TimeSpan = Nothing
Dim timeDiff As TimeSpan = Nothing
Dim ex As Exception = Nothing
Dim FileNameWithoutExtension As String = String.Empty
Try
Dim objTh As Thread = Nothing
objTh = New Thread(AddressOf SaveFileByBuffering)
timeStart = DateTime.Now.TimeOfDay
objTh.IsBackground = True
FileNameWithoutExtension = System.IO.Path.GetFileName(FldUploadThreading.FileName)
objTh.Start("New_" + FileNameWithoutExtension)
objTh.Name = "ARAThreadFileBuffer"
objTh.Join()
timeEnd = DateTime.Now.TimeOfDay
timeDiff = timeEnd - timeStart
Catch exThAbort As ThreadAbortException
ex = exThAbort
Catch exTh As ThreadStartException
ex = exTh
Catch exCommon As Exception
ex = exCommon
End Try
End Sub
Method to be called using thread:
Public Function SaveFileByBuffering(ByVal lstrFilePath As String)
Dim bufferSize As Integer = 512
Dim buffer As Byte() = New Byte(bufferSize - 1) {}
Dim pathUrl As String = ConfigurationManager.AppSettings("strFilePath").ToString()
Dim uploadObj As UploadDetail = New UploadDetail()
uploadObj.IsReady = True
uploadObj.FileName = lstrFilePath
uploadObj.ContentLength = Me.FldUploadThreading.PostedFile.ContentLength
Me.Session("UploadXDetail") = uploadObj
Dim Upload As UploadDetail = DirectCast(Me.Session("UploadXDetail"), UploadDetail)
Dim fileName As String = Path.GetFileName(Me.FldUploadThreading.PostedFile.FileName)
Using fs As New FileStream(Path.Combine(pathUrl, lstrFilePath), FileMode.Create)
While Upload.UploadedLength < Upload.ContentLength
Dim bytes As Integer = Me.FldUploadThreading.PostedFile.InputStream.Read(buffer, 0, bufferSize)
fs.Write(buffer, 0, bytes)
Upload.UploadedLength += bytes
End While
End Using
End Function
There are two issues:
When someone clicks on the same button simultaneously the thread behavior works in a different way, some time page crashes.
When this process I have tested on multi-user environment with 60 users and file size is 25 mb each user the page crashed.
I have to use .NET 3.5 so I cannot use the advanced version of file upload in 2010 or later.
Error : 1-File uploding is more than 15 minutes but still in progress 2- Internet explorer cannot explore the page -Diagnose Internet problems 3- some user get login probem to the server on which the site has hosted
The course I usually take in .NET is to use ThreadPool.QueueUserWorkItem
If you do this with anonymous lambdas, I find it makes the syntax and life rather nice. You can do something like this:
btnUpload.Enabled = False
ThreadPool.QueueUserWorkItem(Sub()
SaveFileByBuffering(FldUploadThreading.FileName)
RaiseEvent EnableButton
End Sub)
With this event handler:
Public Sub EnableButton() Handles EnableButton
If Me.InvokeRequired Then
Me.BeginInvoke(Sub() EnableButton())
Else
btnUpload.Enabled = True
EndIf
EndSub
My .NET is rusty and I don't have a compiler anywhere, but doing something like this should handle most of your issues.
I have a problem when I download a file from a URL (This it's not my main problem), the problem comes after that.
The file that I saved from the URL can be a image, doc, PDF or ZIP.
Exists some method to know the file type when the path doesn't has the extension?
Or identify the file type from an stream?
I'm working with Visual Studio 2010 Express Edition - Framework.Net 3.5 - A Window App
Public Function DownloadFile_FromURL(ByVal URL As String, ByVal DestinationPath As String) As Boolean
DownloadFile_FromURL = False
Try
Dim vRequest As Net.HttpWebRequest
Dim vResponse As Net.HttpWebResponse
vRequest = Net.WebRequest.Create(New Uri(URL))
vRequest.Method = "GET"
vRequest.AllowAutoRedirect = True
vRequest.UseDefaultCredentials = True
vResponse = vRequest.GetResponse
If vResponse.ContentLength <> -1 Then
Dim vLen As Long = vResponse.ContentLength
Dim vWriteStream As New IO.FileStream(DestinationPath, IO.FileMode.CreateNew)
Dim vStream As IO.Stream = vResponse.GetResponseStream()
Dim vReadBytes() As Byte = New Byte(255) {}
Dim vCount As Integer = vStream.Read(vReadBytes, 0, vReadBytes.Length)
While vCount > 0
vWriteStream.Write(vReadBytes, 0, vCount)
vCount = vStream.Read(vReadBytes, 0, vReadBytes.Length)
End While
vWriteStream.Flush() : vWriteStream.Close()
vResponse.Close() : vRequest = Nothing : GCcleaner()
Dim v = System.IO.Path.GetExtension(DestinationPath)
DownloadFile_FromURL = True
End If
Catch ex As Exception
Throw New Exception(ex.mc_GetAllExceptions)
End Try
End Function
If you are using WebRequest for the download.
Dim uri As String = "http://domain.com/resource"
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(uri), HttpWebRequest)
request.Method = "GET"
Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
Dim contentType = response.ContentType
' this will have the content type/ file type
You can now have a routine to save the file using a particular extension depending on the content type. for instance a content type of "image/jpeg" can be saved as *.jpg
For an Image you can load it into an Image() object and see if it throws an OutOfMemoryException -- not an image.
PDF you can read the first few bytes of it (the PDF file type info is stored there, though not sure exactly what it is at the moment).
ZIP and DOC I'm not sure about.
If you are using WebRequests you can grab the content type of the response stream. More information about the MIME/content types here: http://msdn.microsoft.com/en-us/library/ms775147.aspx
Imports System.IO
Imports System.Threading
Imports System.Net.Sockets
Imports System.Text
Module Module1
<MTAThread()>
Sub Main()
Dim maxthreads As Integer = 10
ThreadPool.SetMaxThreads(maxthreads, maxthreads)
Dim serverlist() As String = File.ReadAllLines("final.txt")
Dim servernum As New CountdownEvent(serverlist.GetUpperBound(0) + 1)
For Each a In serverlist
Dim args(1) As Object
args(0) = a
args(1) = servernum
ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf ping), args)
Next
servernum.Wait()
Console.ReadLine()
End Sub
Public Sub ping(ByVal server() As Object)
Dim serverinfo() As String = Split(server(0), ",")
Dim socketclient As New TcpClient
Console.WriteLine(serverinfo(0))
Try
socketclient.Connect(serverinfo(0), serverinfo(1))
Catch ex As Exception
If socketclient.Connected = False Then
Throw New Exception("Server Offline")
Else
Throw New Exception("Unknown Error Occured")
End If
End Try
Dim stream As NetworkStream = socketclient.GetStream
stream.ReadTimeout = 1000
stream.WriteTimeout = 1000
Dim sendBytes As [Byte]() = {&HFE}
stream.Write(sendBytes, 0, sendBytes.Length)
Dim bytes(421) As Byte
stream.Read(bytes, 0, CInt(421))
socketclient.Close()
Dim trimbytes(bytes.Length - 3) As Byte
Array.Copy(bytes, 3, trimbytes, 0, bytes.Length - 3)
Dim returndata As String = Encoding.BigEndianUnicode.GetString(trimbytes)
Dim sb As New System.Text.StringBuilder
For i As Integer = 1 To 241 Step 2
If trimbytes(i) <> 0 Then
sb.Append(ChrW(BitConverter.ToInt16(trimbytes, i)))
End If
Next i
Dim message() As String = sb.ToString.Split("ยง")
'Write processed server information to the console
Console.WriteLine("Received -->" & message(1) & " " & message(2) & " | " & serverinfo(0))
server(1).signal()
End Sub
End Module
Above is my code, what I basically want to do is to get two (or more) ips from the text file, connect to both of them via TCP on two different threads and get the responses from the servers.
Now, the problem here is that; for whatever reason one of the two servers get a response but the other doesn't. I've made sure both of the servers are online and accepting connections. I've also tried having the same server on both lines of the text file so that the same ip is being used on both threads but this doesn't work either. Even when the two ips are the same, one gets a response but not the other; which must mean I'm doing something horribly wrong.
When I use one server rather than 1+ everything works perfectly and I get the response I'm looking for, anything more than 1 and it looks like the first thread that runs produces the response but not anything that runs later. Any help will be appreciated.
I have also tried using Paarallel.For but ended up running into the same issue.
Parallel.ForEach(serverlist, _
Sub(currentElement)
ping(currentElement)
End Sub)