I've got a problem with my VB.Net-application. When i'm trying to send a request to http://ask.fm without proxy, i got the following response in fiddler: HTTP/1.1 502 Fiddler - DNS Lookup Failed
What does this mean? And why is there no error when using a proxy server for the request?
I send the request with the following code:
Public Function GetRequest(ByVal url As String, ByVal referer As String, ByVal cookie As CookieContainer, Optional proxy As String = "") As String
Dim request As HttpWebRequest
Dim response As HttpWebResponse
Try
Dim cookies As String = ""
request = CType(HttpWebRequest.Create(url), HttpWebRequest)
request.CookieContainer = cookie
request.ServicePoint.Expect100Continue = False
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0"
If referer <> "" Then
request.Referer = referer
End If
request.Timeout = 10000
If proxy <> "" Then
request.Proxy = New WebProxy(proxy)
End If
response = CType(request.GetResponse(), HttpWebResponse)
cookies = response.GetResponseHeader("Set-Cookie")
If url.Contains("ask.fm/") Then
If New Regex("l=.*?; domain=ask.fm; path=/;").Match(cookies).Value = "" Then
Return "ERROR"
End If
End If
Dim source As String = New StreamReader(response.GetResponseStream()).ReadToEnd()
If source.Contains("No robots allowed!") Then
source = "ERROR"
End If
If source = "" Then
source = "ERROR"
End If
Return source
Catch ex As Exception
Return "ERROR"
End Try
End Function
Are you using Fiddler to inspect the traffic? If so, make sure Fiddler can redirect your traffic correctly.
Go to Help > About Fiddler and make sure your Gateway is correct (either set to the proxy you use for accessing internet or equal to No Gateway if you can access directly).
Related
I want to check if URL has valid page (not 404, just 200).
Code I've tried:
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(myurl), HttpWebRequest)
request.KeepAlive = True
Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
If response.StatusCode = HttpStatusCode.OK Then
MsgBox("OK")
End If
Yet every URL that I enter leaves OK response, even if I enter http://mywebsite.com/blahblah.
It's not same on all websites (works fine with example.com), but it doesn't work on my website. Why?
In my browser I see 404 page, but code says it's OK.
Edit: Just to mention that my website has Cloudflare on.
Try this out and see if it work's for you... As mentioned above in my comment's:
A 404 just means the page isn't found on the server, status will return ok even if a page isn't found and the server was reached and responded
Public Class WebPage
Public Property PageSource As String = String.Empty
Public Property Status As HttpStatusCode = HttpStatusCode.NotFound
Public Property WebError As String = String.Empty
End Class
Public Shared Function GetWebPage(ByVal Website As String) As WebPage
Dim web As New WebPage() With {.Status = HttpStatusCode.OK}
Try
Using source As New System.Net.WebClient()
web.PageSource = source.DownloadString(Website)
End Using
Return web
Catch exweb As WebException
If exweb.Status = WebExceptionStatus.ProtocolError AndAlso exweb.Message.Contains("404") Then
web.Status = HttpStatusCode.NotFound
Else
web.Status = HttpStatusCode.BadRequest
End If
web.WebError = exweb.Message
Catch ex As Exception
web.Status = HttpStatusCode.NotFound
web.WebError = ex.Message
End Try
Return web
End Function
Usage Example
Dim webObj As WebPage = GetWebPage("THESITE")
If Not String.IsNullOrEmpty(webObj.WebError) Then
MessageBox.Show(webObj.WebError)
ElseIf webObj.Status = HttpStatusCode.OK Then
MessageBox.Show("OK")
End If
This will do what you want.
Function URLExists(url As String) As Boolean
Dim Request As Object
Dim ff As Integer
Dim rc As Variant
On Error GoTo EndNow
Set Request = CreateObject("WinHttp.WinHttpRequest.5.1")
With Request
.Open "GET", url, False
.send
rc = .StatusText
End With
Set Request = Nothing
If rc = "OK" Then URLExists = True
Exit Function
EndNow:
End Function
I am trying to create a Betfair app written in VB.NET using Betfair NG API.
I am trying to request the navigation menu in VB.NET utilizing the following instructions, here, but I get error 400 Bad request. Could someone send me a code for doing so?
Sub GetNavigatorManu(ByVal SessToken As String, ByVal AppKey As String)
Dim request As HttpWebRequest = Nothing
Dim response As HttpWebResponse = Nothing
Dim strResponseStatus As String = ""
Try
request = WebRequest.Create(New Uri("https://api.betfair.com/exchange/betting/rest/v1/en/navigation/menu.json"))
request.Accept = "application/json"
request.Method = "GET"
request.KeepAlive = True
request.ContentType = "application/json"
request.AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate
request.Headers.Add("Accept-Encoding", "gzip,deflate")
request.Headers.Add("X-Authentication", SessToken)
request.Headers.Add("X-Application", AppKey)
response = request.GetResponse()
Display the status below
strResponseStatus = "Status is " & CType(response, HttpWebResponse).StatusCode
Catch ex As Exception
MsgBox("CreateRequest Error" & vbCrLf & ex.Message)
End Try
MsgBox(strResponseStatus)
End Sub
I'm trying to connect to a third party website, and then get the source code.
It worked well for me, and after some time I tried sign in again and then i got an error message:
"{" The server committed a protocol violation. Section = ResponseStatusLine "}.
After quick search on google i found out that i sholud to add to "app.config" the following entry: <httpWebRequest useUnsafeHeaderParsing = true/>
After that it worked fine. But, I get the source code as a guset, and not as "connected User".
I tried another site that does not require "UseUnsafeHeaderParsing = true" and it worked well.
It looks like "UseUnsafeHeaderParsing = true" disruptive cookies?
*Sorry for my english, this is not my native language.
this is my code:
Private siteCookies As New Net.CookieContainer()
Function siteRequest(url As String, ByVal Method As String, Optional ByVal data As String = Nothing) As String
Static soucrecode As String = Nothing
Const UserAgent As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
Try
Dim Request As Net.HttpWebRequest = Net.WebRequest.Create(url)
Request.Accept = "*/*"
Request.Timeout = 10000
Request.Method = Method
Request.UserAgent = UserAgent
Request.AllowAutoRedirect = True
Request.CookieContainer = siteCookies
If Request.Method = "POST" AndAlso data IsNot Nothing Then
Dim postBytes() As Byte = New UTF8Encoding().GetBytes(data)
Request.ContentType = "application/x-www-form-urlencoded"
Request.ContentLength = postBytes.Length
Request.GetRequestStream().Write(postBytes, 0, postBytes.Length)
End If
Dim Response As Net.HttpWebResponse = Request.GetResponse()
soucrecode = New IO.StreamReader(Response.GetResponseStream).ReadToEnd()
Response.Close()
Catch e As Exception
Console.WriteLine(e.Message)
End Try
Return soucrecode
End Function
using:
txtLoginSoucre.Text = siteRequest("http://www.SomeSite.com/login.php?do=login", "Post", "username=myUser&password=MyPass") ' Login to site
txtSoucre.Text siteRequest("http: //www.SomeSite.com", "Get") 'Grab soucrecode
Just add
Request.KeepAlive = False
i was inspired by Awesomium and im trying to use it as my client side app for my web application using vb.net . im trying to do an uploader module where our clinets can upload any files to our server . i used HttpWebRequest for uploading files which is working fine. but only problem is how to set the session created by Awesomium when i logged on to my web application to httpwebrequest . or is there any other way in awesomium itself to upload files to the server (ie php server ) .
Please apologies im not good in English.
below is the code im using for upload
Dim filepath As String = Path 'Path to file on local machine
Dim url As String = "http://xxxxx.com/uploadscanfile.php"
Dim boundary As String = IO.Path.GetRandomFileName
ImageRandomName.Add(IO.Path.GetFileName(filepath))
Dim header As New System.Text.StringBuilder()
header.AppendLine("--" & boundary)
header.Append("Content-Disposition: form-data; name=""file"";")
header.AppendFormat("filename=""{0}""", IO.Path.GetFileName(filepath))
header.AppendLine()
header.AppendLine("Content-Type: application/octet-stream")
header.AppendLine()
Dim headerbytes() As Byte = System.Text.Encoding.UTF8.GetBytes(header.ToString)
Dim endboundarybytes() As Byte = System.Text.Encoding.ASCII.GetBytes(vbNewLine & "--" & boundary & "--" & vbNewLine)
Dim req As Net.HttpWebRequest = Net.HttpWebRequest.Create(url)
req.ContentType = "multipart/form-data; boundary=" & boundary
req.ContentLength = headerbytes.Length + New IO.FileInfo(filepath).Length + endboundarybytes.Length
req.AllowAutoRedirect = True
req.Timeout = -1
req.KeepAlive = True
req.AllowWriteStreamBuffering = False
req.Method = "POST"
Dim s As IO.Stream = req.GetRequestStream
s.Write(headerbytes, 0, headerbytes.Length)
Dim filebytes() As Byte = My.Computer.FileSystem.ReadAllBytes(filepath)
s.Write(filebytes, 0, filebytes.Length)
s.Write(endboundarybytes, 0, endboundarybytes.Length)
s.Close()
Typically, session info is stored in the cookies. So, you will first need to send the log in data (and receive a reply). Also, when receiving the reply, you need to set a CookieContainer in the CookieContainer property and you will reuse it later. Only then, you can send the form data and upload the file, but make sure to set the CookieContainer to the CookieContainer that has the log in cookies when uploading.
A good way to see all the request/responses that go through to/from the server when loging in, I suggest you use Fiddler it is a very useful tool that monitors all the requests that you make. Note the headers, data that gets send and anything else that you may find useful.
More code:
Here is the part that logs you in: The post string that you send needs to contain the username and password. (and any other info that may be required).
Dim CookieJar As New CookieContainer() 'The CookieContainer that will keep all the cookies.
'DO NOT CLEAR THIS BETWEEN REQUESTS! ONLY CLEAR TO "Log Out".
Dim req As HttpWebRequest = HttpWebRequest.Create("<login URL goes here>")
req.Method = "POST"
req.Accept = "text/html, application/xhtml+xml, */*" 'This may be a bit different in your case. Refer to what Fiddler will say.
req.CookieContainer = CookieJar
req.ContentLength = post_str.Length
req.ContentType = "application/x-www-form-urlencoded" 'Also, any useragent will do.
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1"
req.Headers.Add("Accept-Language", "en-US,en;q=0.7,ru;q=0.3")
req.Headers.Add("Accept-Encoding", "gzip, deflate")
req.Headers.Add("DNT", "1") 'Make sure to add all headers that you found using Fiddler!
req.Headers.Add("Pragma", "no-cache")
Dim RequestWriter As New IO.StreamWriter(req.GetRequestStream())
RequestWriter.Write(post_str) 'Write the post string that contains the log in, password, etc.
RequestWriter.Close()
RequestWriter.Dispose()
Dim ResponceReader As New IO.StreamReader(req.GetResponse().GetResponseStream())
Dim ResponceData As String = ResponceReader.ReadToEnd()
ResponceReader.Close()
ResponceReader.Dispose()
req.GetResponse.Close()
'In the long run, you can check the ResponceData to verify that the log in was successful.
Here is where the CookieJar gets used to sent a request:
post_str = "" 'What needs to be sent goes in this variable.
req = HttpWebRequest.Create("<page to send request to goes here>")
req.Method = "POST"
req.Accept = "text/html, application/xhtml+xml, */*" 'May be different in your case
req.CookieContainer = CookieJar 'Please note: this HAS to be the same CookieJar as you used to login.
req.ContentLength = post_str.Length
req.ContentType = "application/x-www-form-urlencoded"
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1"
req.Headers.Add("Accept-Language", "en-US,en;q=0.7,ru;q=0.3")
req.Headers.Add("Accept-Encoding", "gzip, deflate")
req.Headers.Add("DNT", "1") 'Add all headers here.
req.Headers.Add("Pragma", "no-cache")
RequestWriter = New IO.StreamWriter(req.GetRequestStream())
RequestWriter.Write(post_str)
RequestWriter.Close()
RequestWriter.Dispose()
ResponceReader = New IO.StreamReader(req.GetResponse().GetResponseStream())
ResponceData = ResponceReader.ReadToEnd()
ResponceReader.Close()
ResponceReader.Dispose()
req.GetResponse.Close()
'You may want to read the ResponceData.
Hope this helps.
So i have this class that I use for HTTPwebrequests. In this instance of my program I'm posting some data as well as a cookie. Now heres the really wierd part. I use fiddler to sniff headers and monitor the traffic of my program in testing. If fiddler is open this function works just fine. If fidler is closed whenever I go to post anydata the operation times out. It always does this. No matter what I do. Can any one shed some light on this as I'm pulling my hair out in frustration.
Public Function Send(ByVal URL As String, _
Optional ByVal PostData As String = "", _
Optional ByVal Method As HTTPMethod = HTTPMethod.HTTP_GET, _
Optional ByVal ContentType As String = "", Optional ByVal Refer As String = "")
Dim Request As HttpWebRequest = WebRequest.Create(URL)
Dim Response As HttpWebResponse
System.Net.ServicePointManager.Expect100Continue = False
Dim SW As StreamWriter
Dim SR As StreamReader
Dim ResponseData As String
Dim a As New CookieContainer()
' Request.Proxy = New WebProxy("173.234.250.164", 3128)
' Prepare Request Object
Request.Method = Method.ToString().Substring(5)
Request.CookieContainer = a
' Set form/post content-type if necessary
If (Method = HTTPMethod.HTTP_POST AndAlso PostData <> "" AndAlso ContentType = "") Then
ContentType = "application/x-www-form-urlencoded"
End If
'Set User Agent
Request.UserAgent = ("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB6.6)")
'Set Refer
If (Refer <> "") Then
Request.Referer = Refer
End If
' Set Content-Type
If (ContentType <> "") Then
Request.ContentType = ContentType
Request.ContentLength = PostData.Length
End If
'Set Cookie If Given
If (Cookie <> "") Then
Request.Headers.Add("cookie", Cookie)
End If
' Send Request, If Request
If (Method = HTTPMethod.HTTP_POST) Then
Try
Debug.Print("Inside Post")
SW = New StreamWriter(Request.GetRequestStream())
SW.Write(PostData)
Debug.Print("Wrote Post Data")
Catch Ex As Exception
Throw Ex
Finally
SW.Close()
End Try
End If
' Receive Response
Try
Response = Request.GetResponse()
For Each cook As Cookie In Response.Cookies
Cookie = Cookie & cook.ToString() & ";"
Next
Debug.Print(Cookie)
SR = New StreamReader(Response.GetResponseStream())
ResponseData = SR.ReadToEnd()
Catch Wex As System.Net.WebException
SR = New StreamReader(Wex.Response.GetResponseStream())
ResponseData = SR.ReadToEnd()
Throw New Exception(ResponseData)
Finally
SR.Close()
End Try
Return ResponseData
End Function
You're never disposing the HttpWebResponse - which means the connection is going to be held open. You should use a Using statement for Response. (I'd also suggest that you declare variables as late as possible, rather than declaring everything at the top of the method... and that you use Using statements everywhere instead of closing things in Finally blocks.
Note that you're also closing SR in a finally block when it may not be set - if some exception you weren't expecting is thrown, your attempt to close SR may throw a NullReferenceException. Personally I'd treat the two StreamReaders (one in the success case and one in the failure case) separately.