I have around 40 values in my database. I am checking every single url if it is online or offline. This is the function I am using:
Public Function CheckAddress(ByVal URL As String) As Boolean
Try
Dim request As WebRequest = WebRequest.Create(URL)
Dim response As WebResponse = request.GetResponse()
Catch ex As Exception
Return False
End Try
Return True
End Function
But this is taking too much time. How can we load the page faster?
I added timeout to WebRequest. Like:
Dim request As WebRequest = WebRequest.Create(URL)
request.Timeout = 50
Maybe you can check for headers only, via the HEAD method:
Public Function CheckAddress(ByVal URL As String) As Boolean
Try
Dim request As WebRequest = WebRequest.Create(URL)
request.Method = "HEAD" ' only headers!
Dim response As WebResponse = request.GetResponse()
If response IsNot Nothing AndAlso _
response.Headers IsNot Nothing AndAlso _
response.Headers.count > 0
...
' check if you get anything
Return True
else
Return false
end if
Catch ex As Exception
Return False
End Try
End Function
This may be quicker.
Related
I am getting TWO exceptions. The first one in the regular code and the second into the exception handling block.
I have this function (below) that supposed to change user settings in a Web API using PUT method. Everything runs fine until the point that I try to get a response from the web request. Then it raises the first exception (unknown) and when I try to handle the error I get a second exception "Object reference not set to an instance of an object".
As far as I know normally what will cause this error is the attempt of getting a response of an unassigned web request, but in this case it IS assigned and the Uri is valid. All the credentials for authorization are correct too.
This is my code:
Try
Dim webRequest As System.Net.HttpWebRequest
webRequest = System.Net.HttpWebRequest.Create(InternalSettings.APIurl + "/config")
webRequest.Headers("Authorization") = InternalSettings.APIpwd
webRequest.Headers("API-Application-Key") = InternalSettings.APIkey
webRequest.Method = "PUT"
webRequest.ContentType = "application/x-www-form-urlencoded"
Dim postData As String = ""
postData += "SaveAllCustomersData=false"
Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postData)
Dim dataStream As Stream = webRequest.GetRequestStream()
dataStream.Write(byteArray, 0, byteArray.Length)
dataStream.Close()
'The next line will raise an unknown exception
Dim myHttpWebResponse As HttpWebResponse = webRequest.GetResponse
Dim responseReader As StreamReader = New StreamReader(myHttpWebResponse.GetResponseStream())
Dim responseData As String = responseReader.ReadToEnd()
responseReader.Close()
webRequest.GetResponse().Close()
Catch ex As WebException
'The next line will raise a "Object reference not set to an instance of an object" exception
Dim resp = New StreamReader(ex.Response.GetResponseStream()).ReadToEnd()
Dim obj = JsonConvert.DeserializeObject(resp)
Return False
End Try
If you are using basic authentification you can try something like this:
Dim strUrl As String = InternalSettings.APIurl + "/config"
Dim request As WebRequest = DirectCast(WebRequest.Create(strUrl), HttpWebRequest)
Dim strResponse As String = ""
Dim byteAuth() As Byte = Encoding.UTF8.GetBytes(InternalSettings.APIkey & ":" & InternalSettings.APIpwd)
Dim strPost As String = "SaveAllCustomersData=false"
Dim bytePost() As Byte = Encoding.UTF8.GetBytes(strPost)
request.ContentType = "application/x-www-form-urlencoded"
request.Method = "PUT"
request.Headers.Add("Authorization", "Basic " & Convert.ToBase64String(byteAuth))
Using sw = New StreamWriter(request.GetRequestStream())
sw.Write(bytePost, 0, bytePost.Length)
sw.Flush()
Using response As HttpWebResponse = request.GetResponse()
Using sr = New StreamReader(response.GetResponseStream())
strResponse = sr.ReadToEnd()
End Using
End Using
End Using
I want to check if URL has valid page (not 404, just 200).
Code I've tried:
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(myurl), HttpWebRequest)
request.KeepAlive = True
Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
If response.StatusCode = HttpStatusCode.OK Then
MsgBox("OK")
End If
Yet every URL that I enter leaves OK response, even if I enter http://mywebsite.com/blahblah.
It's not same on all websites (works fine with example.com), but it doesn't work on my website. Why?
In my browser I see 404 page, but code says it's OK.
Edit: Just to mention that my website has Cloudflare on.
Try this out and see if it work's for you... As mentioned above in my comment's:
A 404 just means the page isn't found on the server, status will return ok even if a page isn't found and the server was reached and responded
Public Class WebPage
Public Property PageSource As String = String.Empty
Public Property Status As HttpStatusCode = HttpStatusCode.NotFound
Public Property WebError As String = String.Empty
End Class
Public Shared Function GetWebPage(ByVal Website As String) As WebPage
Dim web As New WebPage() With {.Status = HttpStatusCode.OK}
Try
Using source As New System.Net.WebClient()
web.PageSource = source.DownloadString(Website)
End Using
Return web
Catch exweb As WebException
If exweb.Status = WebExceptionStatus.ProtocolError AndAlso exweb.Message.Contains("404") Then
web.Status = HttpStatusCode.NotFound
Else
web.Status = HttpStatusCode.BadRequest
End If
web.WebError = exweb.Message
Catch ex As Exception
web.Status = HttpStatusCode.NotFound
web.WebError = ex.Message
End Try
Return web
End Function
Usage Example
Dim webObj As WebPage = GetWebPage("THESITE")
If Not String.IsNullOrEmpty(webObj.WebError) Then
MessageBox.Show(webObj.WebError)
ElseIf webObj.Status = HttpStatusCode.OK Then
MessageBox.Show("OK")
End If
This will do what you want.
Function URLExists(url As String) As Boolean
Dim Request As Object
Dim ff As Integer
Dim rc As Variant
On Error GoTo EndNow
Set Request = CreateObject("WinHttp.WinHttpRequest.5.1")
With Request
.Open "GET", url, False
.send
rc = .StatusText
End With
Set Request = Nothing
If rc = "OK" Then URLExists = True
Exit Function
EndNow:
End Function
I am not familiar a lot with Vb.Net, but I try to tweak something on an existent project. I have a cURL that try to implement on Vb.Net. I found different answers here and in other forums, but this way is the one I managed to reach
Private Function SendRequest(uri As Uri, jsonDataBytes As Byte(), contentType As String, method As String) As String
Dim req As WebRequest = WebRequest.Create(uri)
req.ContentType = contentType
req.Method = method
req.ContentLength = jsonDataBytes.Length
Dim stream = req.GetRequestStream()
stream.Write(jsonDataBytes, 0, jsonDataBytes.Length)
stream.Close()
Dim response = req.GetResponse().GetResponseStream()
Dim reader As New IO.StreamReader(response)
Dim res = reader.ReadToEnd()
reader.Close()
response.Close()
Return res
End Function
Dim postData As String = String.Format("text={0}", title)
Dim data = Encoding.UTF8.GetBytes(postData)
Dim uri = New Uri("https://.....")
Dim slackResponse = SendRequest(uri, data, "application/json", "POST")
And this is the error I get:
Exception Details: System.Net.WebException: The remote server returned an error: (400) Bad Request.
If I comment out the SendRequest function, I got an error during calling that, so I guess it is on that part.
Not able to debug more. Any ideas?
In order to debug more, you need to catch the WebException. WebExceptions have a response object that may contain more information.
Dim response as WebResponse
Try
response = req.GetResponse().GetResponseStream()
Catch ex As Net.WebException
If ex.Response IsNot Nothing Then
response = ex.Response
End If
End Try
This could be a problem with your URL parameters but many sites also return this class of error when there is a problem in the json content of the request.
I solved this long time ago by adding.
request.ContentType = "application/x-www-form-urlencoded"
Can anybody tell me why the function below always returns true, even if the FTP directory in question does not exist?
The value of directoryURL which I pass in is of the form:
ftp://ip_address/directory/subdirectory/
and has a trailing forward slash.
Public Function DoesDirectoryExist(directoryUrl As String) As Boolean
' Check that the target URL is properly formatted
If Not directoryUrl.StartsWith("ftp://") Then directoryUrl = "ftp://" & directoryUrl
' Create a web request
Dim request As FtpWebRequest = DirectCast(WebRequest.Create(directoryUrl), FtpWebRequest)
request.Credentials = New NetworkCredential(_userName, _password)
request.Method = WebRequestMethods.Ftp.ListDirectory
' Try and list the contents of the directory
Try
Using response As FtpWebResponse = DirectCast(request.GetResponse(), FtpWebResponse)
' We have been succesful so the directory exists
Return True
End Using
Catch ex As WebException
Dim response As FtpWebResponse = DirectCast(ex.Response, FtpWebResponse)
If response.StatusCode = FtpStatusCode.ActionNotTakenFileUnavailable Then
Return False
Else
Throw New ApplicationException("Unable to determine if FTP directory exists.")
End If
End Try
End Function
Weird. This works for me (I don't cast the request but I guess that shouldn't matter). This is the code I usually rely on:
Dim response As FtpWebResponse = request.GetResponse()
Using (response)
found = True
End Using
The alternative you have is reading the list of directories:
Using sr As New System.IO.StreamReader(response.GetResponseStream())
Using sw As New System.IO.StreamWriter("tempfile", False)
sw.Write(sr.ReadToEnd())
End Using
End Using
In the worst scenario, it should help you to tackle the problem (e.g., it always founds a directory called "ghost", which you might use to trigger the not-found).
Method 1
Public Function DirectoryExists(directory As String) As Boolean
Dim directoryExists__1 As Boolean
Dim request = DirectCast(WebRequest.Create(directory), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.ListDirectory
request.Credentials = New NetworkCredential("user", "pass")
Try
Using request.GetResponse()
directoryExists__1 = True
End Using
Catch generatedExceptionName As WebException
directoryExists__1 = False
End Try
Return directoryExists__1
End Function
Method 2
If Not DirectoryExists("ftp://" + FTPSettings.IP + "/" + lo_ScreenShotPath) Then
reqFTP = DirectCast(FtpWebRequest.Create(New Uri("ftp://" + FTPSettings.IP + "/" + lo_ScreenShotPath)), FtpWebRequest)
End If
i hope may i help this...
So i have this class that I use for HTTPwebrequests. In this instance of my program I'm posting some data as well as a cookie. Now heres the really wierd part. I use fiddler to sniff headers and monitor the traffic of my program in testing. If fiddler is open this function works just fine. If fidler is closed whenever I go to post anydata the operation times out. It always does this. No matter what I do. Can any one shed some light on this as I'm pulling my hair out in frustration.
Public Function Send(ByVal URL As String, _
Optional ByVal PostData As String = "", _
Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
Optional ByVal ContentType As String = "", Optional ByVal Refer As String = "")
Dim Request As HttpWebRequest = WebRequest.Create(URL)
Dim Response As HttpWebResponse
System.Net.ServicePointManager.Expect100Continue = False
Dim SW As StreamWriter
Dim SR As StreamReader
Dim ResponseData As String
Dim a As New CookieContainer()
' Request.Proxy = New WebProxy("173.234.250.164", 3128)
' Prepare Request Object
Request.Method = Method.ToString().Substring(5)
Request.CookieContainer = a
' Set form/post content-type if necessary
If (Method = HTTPMethod.HTTP_POST AndAlso PostData <> "" AndAlso ContentType = "") Then
ContentType = "application/x-www-form-urlencoded"
End If
'Set User Agent
Request.UserAgent = ("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB6.6)")
'Set Refer
If (Refer <> "") Then
Request.Referer = Refer
End If
' Set Content-Type
If (ContentType <> "") Then
Request.ContentType = ContentType
Request.ContentLength = PostData.Length
End If
'Set Cookie If Given
If (Cookie <> "") Then
Request.Headers.Add("cookie", Cookie)
End If
' Send Request, If Request
If (Method = HTTPMethod.HTTP_POST) Then
Try
Debug.Print("Inside Post")
SW = New StreamWriter(Request.GetRequestStream())
SW.Write(PostData)
Debug.Print("Wrote Post Data")
Catch Ex As Exception
Throw Ex
Finally
SW.Close()
End Try
End If
' Receive Response
Try
Response = Request.GetResponse()
For Each cook As Cookie In Response.Cookies
Cookie = Cookie & cook.ToString() & ";"
Next
Debug.Print(Cookie)
SR = New StreamReader(Response.GetResponseStream())
ResponseData = SR.ReadToEnd()
Catch Wex As System.Net.WebException
SR = New StreamReader(Wex.Response.GetResponseStream())
ResponseData = SR.ReadToEnd()
Throw New Exception(ResponseData)
Finally
SR.Close()
End Try
Return ResponseData
End Function
You're never disposing the HttpWebResponse - which means the connection is going to be held open. You should use a Using statement for Response. (I'd also suggest that you declare variables as late as possible, rather than declaring everything at the top of the method... and that you use Using statements everywhere instead of closing things in Finally blocks.
Note that you're also closing SR in a finally block when it may not be set - if some exception you weren't expecting is thrown, your attempt to close SR may throw a NullReferenceException. Personally I'd treat the two StreamReaders (one in the success case and one in the failure case) separately.