I use MS Visual Studio 2017 in Windows 10 and tried following httpclient by VB.net in order to get html of, for example, yahoo.com. I could get Header OK status 200 while its contents consists of several funny letters and finally resulsted in empty when converting to string.
Would you please advise how to receive contents by means of httpclient?
I tried variety sets of header which are not a cause of the problem. I assume the cause exists in how to convert the contents or I may need to access again the site after receiving a header in order to receive the content???.
Dim url As String = "https://yahoo.com"
Dim httpclienthandler0 = New HttpClientHandler()
httpclienthandler0.UseCookies = True
httpclienthandler0.SslProtocols = SecurityProtocolType.Tls Or SecurityProtocolType.Tls11 Or SecurityProtocolType.Tls12 Or SecurityProtocolType.Tls13
httpclienthandler0.ServerCertificateCustomValidationCallback = AddressOf OnRemoteCertificateValidationCallback
Dim httpclient0 As New HttpClient(httpclienthandler0)
httpclient0.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9")
httpclient0.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br")
httpclient0.DefaultRequestHeaders.Add("Accept-Language", "en-us;q=0.7,en;q=0.3")
httpclient0.DefaultRequestHeaders.Add("Cache-Control", "max-age=0")
httpclient0.DefaultRequestHeaders.Add("Connection", "keep-alive")
httpclient0.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1")
httpclient0.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36")
httpclient0.Timeout = TimeSpan.FromSeconds(10.0)
Try
Dim res As HttpResponseMessage = Await httpclient0.GetAsync(New Uri(url))
res.EnsureSuccessStatusCode()
Dim responseBody As String = Await res.Content.ReadAsStringAsync()
Dim enc As System.Text.Encoding = System.Text.Encoding.GetEncoding("utf-8")
Dim html As String = String.Empty
Using stream = Await res.Content.ReadAsStreamAsync()
Using reader = DirectCast(New IO.StreamReader(stream, enc, True), IO.TextReader)
html = Await reader.ReadToEndAsync()
End Using
End Using
Catch ex As HttpRequestException
Dim ex1 As Exception = ex
While (ex1 IsNot Nothing)
Console.WriteLine("Ex Message: {0} ", ex1.Message)
ex1 = ex1.InnerException
End While
Catch ex As TaskCanceledException
Dim ex2 As Exception = ex
Console.WriteLine(vbCr + "Timeout!)
Console.WriteLine("Ex MEssage: {0} ", ex2.Message)
End Try
Following codes derived from above are specific code that gets empty contents.
Dim responseBody As String = Await res.Content.ReadAsStringAsync()
and I tried next code instead of responseBody.
Using stream = Await res.Content.ReadAsStreamAsync()
Using reader = DirectCast(New IO.StreamReader(stream, enc, True), IO.TextReader)
html = Await reader.ReadToEndAsync()
End Using
End Using
However both code resulted in empty.
Thank you indeed for your time to read and answer.
Related
so i am trying to login to a website using httpwebrequest. the post data i got from a http debugger is
code i am trying is:
Dim postData As String = "securitycheck=85b39cc89f04bc1612ce9d0c384b39ca&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=jawademail&pass=jawadpass"
Dim tempCookies As New CookieContainer
Dim encoding As New UTF8Encoding
Dim byteData As Byte() = encoding.GetBytes(postData)
Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.dreamstime.com/securelogin.php"), HttpWebRequest)
postReq.Method = "POST"
postReq.KeepAlive = True
postReq.CookieContainer = tempCookies
postReq.ContentType = "application/x-www-form-urlencoded"
postReq.Referer = "https://www.dreamstime.com/login.php"
postReq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)"
postReq.ContentLength = byteData.Length
Dim postreqstream As Stream = postReq.GetRequestStream()
postreqstream.Write(byteData, 0, byteData.Length)
postreqstream.Close()
Dim postresponse As HttpWebResponse
postresponse = DirectCast(postReq.GetResponse(), HttpWebResponse)
tempCookies.Add(postresponse.Cookies)
logincookie = tempCookies
Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
Dim thepage As String = postreqreader.ReadToEnd
RichTextBox1.Text = thepage
thsi code does not seem to post data in website i get referer page code in richtextbox after running the code.
First GET the page, find the "securitycheck" in its source and extract it.
Combine it with the rest of your data then send it with POST.
Ok so I felt like trying:
Dim LoginData As String
Dim LoginCookies As New CookieContainer() 'Move this outside of sub/function so you can use it later
Dim LoginRequest As HttpWebRequest = WebRequest.Create("https://www.dreamstime.com/login.php")
LoginRequest.CookieContainer = LoginCookies
LoginRequest.KeepAlive = True
LoginRequest.AllowAutoRedirect = True
LoginRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0"
Dim LoginResponse As HttpWebResponse = LoginRequest.GetResponse()
Dim LoginResponseRead As StreamReader = New StreamReader(LoginResponse.GetResponseStream())
Using LoginResponseRead
Do
Dim line As String = LoginResponseRead.ReadLine
If line.Contains("var securitycheck=") Then
LoginData = "securitycheck=" & line.Substring(line.IndexOf("=") + 2, line.LastIndexOf("'") - line.IndexOf("=") - 2)
Exit Do
End If
Loop
End Using
Dim byteData As Byte() = Encoding.UTF8.GetBytes(LoginData)
LoginRequest = WebRequest.Create("https://www.dreamstime.com/securelogin.php")
LoginRequest.CookieContainer = LoginCookies
LoginRequest.Method = "POST"
LoginRequest.KeepAlive = True
LoginRequest.ContentType = "application/x-www-form-urlencoded"
LoginRequest.Referer = "https://www.dreamstime.com/login.php"
LoginRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)"
LoginRequest.ContentLength = byteData.Length
Dim postreqstream As Stream = LoginRequest.GetRequestStream()
postreqstream.Write(byteData, 0, byteData.Length)
postreqstream.Close()
LoginResponse = LoginRequest.GetResponse()
LoginResponseRead = New StreamReader(LoginResponse.GetResponseStream())
Dim thepage As String = LoginResponseRead.ReadToEnd
'Now with GET request grab whatever you want, DON'T forget to use cookie.
Result
>>>securitycheck=183d5abdb01f288aacbe5b2893555ec5
Dim email As String = "something"
Dim password As String = "somethingelse"
LoginData &= "&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=" & email & "&pass=" & password
>>>securitycheck=183d5abdb01f288aacbe5b2893555ec5&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=something&pass=somethingelse
There, practically done.
I took a look at this myself and it appears that with every login request a token is sent that identifies your "session", specifically:
securitycheck=85b39cc89f04bc1612ce9d0c384b39ca
This token changes every time you login, and if it isn't valid the site redirects you back to the login page, asking you to login again.
Sites usually do this to prevent Cross-Site Request Forgery (CSRF). This means that you will most likely not be able to login to this site without using an actual web browser.
Here's the code that was tested and it works. It uses System.Net.Http.HttpClient rather WebClient (since it supports concurrent requests). This code is just a model since its main goal is to show the idea how to work with this site. There are additional explanations in comments. You also need to import System.Web dll.
Imports System.Net.Http
Imports System.Web
Imports System.Text.RegularExpressions
Public Class TestForm
Private Const URL_MAIN$ = "https://www.dreamstime.com"
Private Const URL_LOGIN$ = "https://www.dreamstime.com/securelogin.php"
Private Const URL_LOGOUT$ = "https://www.dreamstime.com/logout.php "
Private Const USER_AGENT$ = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
"AppleWebKit/537.36 (KHTML, Like Gecko) " +
"Chrome/68.0.3440.15 Safari/537.36 OPR/55.0.2991.0 " +
"(Edition developer)"
Private Const LOGIN$ = "<USER_NAME>"
Private Const PASS$ = "<USER_PASSWORD>"
Private token$
Private Async Sub OnGo() Handles btnGo.Click
Dim html$
Using client = New HttpClient()
client.DefaultRequestHeaders.Add("User-Agent", USER_AGENT)
Using req = New HttpRequestMessage(HttpMethod.Get, URL_MAIN)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Search for security token
Dim m = Regex.Match(
html,
"<input type=""hidden"" name=""securitycheck"" value=""(?'token'\w+)"">")
If Not m.Success Then
MessageBox.Show("Could not find security token.")
Return
End If
'// Get security token
token = m.Groups("token").Value
'// Try to login.
'// For logging to work, we need to use FormUrlEncodedContent class.
'// Also we need to use it every time we do POST requests.
'// No need for it for GET requests (as long as the HttpClient is the same).
Using req = New HttpRequestMessage(HttpMethod.Post, URL_LOGIN) With
{
.Content = GetFormData()
}
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Go to main page to check we're logged in.
'// "html" variable now MUST contain user's account name.
Using req = New HttpRequestMessage(HttpMethod.Get, URL_MAIN$)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Logout.
'// "html" variable now MUST NOT contain user's account name.
Using req = New HttpRequestMessage(HttpMethod.Get, URL_LOGOUT)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
End Using
End Sub
Function GetFormData() As FormUrlEncodedContent
Return New FormUrlEncodedContent(New Dictionary(Of String, String) From
{
{"securitycheck", token},
{"do_action", "log_into_system"},
{"jump_to", ""},
{"uname", HttpUtility.HtmlEncode(LOGIN)},
{"pass", HttpUtility.HtmlEncode(PASS)}
})
End Function
End Class
I want to take the Flight schedule from the following address:
http://fo-apac.ttinteractive.com/Zenith/FrontOffice/(S(nves1yv4xxoia40cmotixof1))/USBangla/en-GB/BookingEngine/SearchFlights?__cnv=tShqK&json={"BookingPathArguments":null,"OriginDestinations":[{"IsOpen":false,"DataIdOrigin":6337,"DataIdDestination":6707,"DateTime":"2016-04-27T00:00:00.000"}],"TravelerTypes":[{"DataId":1,"TravelerCount":1},{"DataId":2,"TravelerCount":0},{"DataId":3,"TravelerCount":0}],"Currency":{"Code":"BDT"},"PromoCode":null,"DisplayRealAvailability":false,"Visibility":0,"ExtendedSearchDayCount":3}
If you paste the address to browser address bar and use firebug (or any fiddler), you will see this page sends 3 jquery ajax calls to bring the schedule. The following ajax POST actually fetch the schedule.
http://fo-apac.ttinteractive.com/Zenith/FrontOffice/(S(nves1yv4xxoia40cmotixof1))/USBangla/en-GB/FlexibleFlightStaticAjax/FlexibleFlightListLoadSelectedDays?__cnv=mxw0s
PostData : SaleConditionAccepted=false&ExtendedSearchDayCount=3&DoNotCheckCache=false&AlreadyLoggedIn=false&TempDataGuid=nves1yv4xxoia40cmotixof1&CurrencyCode=BDT&FareBasisDataId=&Travelers[0][DataId]=1&Travelers[0][Count]=1&UserSelections[0][SelectedDate]=2016-04-27T00:00:00&UserSelections[0][ReferenceDate]=2016-04-27T00:00:00&UserSelections[0][DataIdOrigin]=6337&UserSelections[0][DataIdDestination]=6707&UserSelections[0][GenericClassDataId]=&UserSelections[0][SelectedSegments]=&JsonPrepareBookingRequest=&PromoCode=
I am sending the request with httpwebrequest, but for unknown reason, I miss the session. I used CookieContainer to keep the cookies. I used the following function to send httprequest:
Public Function GetPostWP(ByVal Url As String, ByVal CkCont As CookieContainer, Optional ByVal PostData As String = "", Optional ByVal refSite As String = "") As String
Dim pStr As String = ""
Try
Dim Http As HttpWebRequest = WebRequest.Create(Url)
If refSite <> "" Then Http.Referer = refSite
Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
Http.CookieContainer = CkCont 'Initial CkCont is Nothing
Http.KeepAlive = True
Http.AllowAutoRedirect = True
'Http.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
Http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
If PostData <> "" Then
Http.Method = "POST"
Http.ContentLength = PostData.Length
Http.ContentType = "application/x-www-form-urlencoded"
Dim PostStream As StreamWriter = New StreamWriter(Http.GetRequestStream())
PostStream.Write(PostData)
PostStream.Close()
End If
Using WebResponse As HttpWebResponse = Http.GetResponse()
Dim responseStream As Stream = WebResponse.GetResponseStream()
If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If
Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
pStr = reader.ReadToEnd()
responseStream.Close()
End Using
tmpCky = CkCont 'tmpCky is a Public CookieContainer Variable to hold cookies for future use.
GetPostWP = pStr
Catch ex As Exception
GetPostWP = "Error : " & ex.Message
End Try
End Function
Dim Cky As New CookieContainer
Dim Txt as String = GetPostWP(PostAddress, Cky, PostData, RefAdd)
Cky = tmpCky
Can anyone analyze the page, please?
I'm trying to connect to a third party website, and then get the source code.
It worked well for me, and after some time I tried sign in again and then i got an error message:
"{" The server committed a protocol violation. Section = ResponseStatusLine "}.
After quick search on google i found out that i sholud to add to "app.config" the following entry: <httpWebRequest useUnsafeHeaderParsing = true/>
After that it worked fine. But, I get the source code as a guset, and not as "connected User".
I tried another site that does not require "UseUnsafeHeaderParsing = true" and it worked well.
It looks like "UseUnsafeHeaderParsing = true" disruptive cookies?
*Sorry for my english, this is not my native language.
this is my code:
Private siteCookies As New Net.CookieContainer()
Function siteRequest(url As String, ByVal Method As String, Optional ByVal data As String = Nothing) As String
Static soucrecode As String = Nothing
Const UserAgent As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
Try
Dim Request As Net.HttpWebRequest = Net.WebRequest.Create(url)
Request.Accept = "*/*"
Request.Timeout = 10000
Request.Method = Method
Request.UserAgent = UserAgent
Request.AllowAutoRedirect = True
Request.CookieContainer = siteCookies
If Request.Method = "POST" AndAlso data IsNot Nothing Then
Dim postBytes() As Byte = New UTF8Encoding().GetBytes(data)
Request.ContentType = "application/x-www-form-urlencoded"
Request.ContentLength = postBytes.Length
Request.GetRequestStream().Write(postBytes, 0, postBytes.Length)
End If
Dim Response As Net.HttpWebResponse = Request.GetResponse()
soucrecode = New IO.StreamReader(Response.GetResponseStream).ReadToEnd()
Response.Close()
Catch e As Exception
Console.WriteLine(e.Message)
End Try
Return soucrecode
End Function
using:
txtLoginSoucre.Text = siteRequest("http://www.SomeSite.com/login.php?do=login", "Post", "username=myUser&password=MyPass") ' Login to site
txtSoucre.Text siteRequest("http: //www.SomeSite.com", "Get") 'Grab soucrecode
Just add
Request.KeepAlive = False
i have a vb.net console application that logged into a website (POST form) by using Webclient:
Dim responsebytes = myWebClient.UploadValues("https:!!xxx.com/mysession/create", "POST", myNameValueCollection)
Last friday this suddenly stopped working, it worked without a problem for about 2-3 years. With Fiddler I got a HTTP 504 error but without Fiddler I got the error message:
The underlying connection was closed: The connection was closed unexpectedly.
I assume that something on the server-side has changed, but I have no influence on that. It's a commercial website, where I want to login automatically on my account to fetch some data.
As Fiddler can't help me much further I decided to built a basic HttpWebRequest example to rule out it was caused by the WebClient.
The example does:
navigate to the homepage of the company and read out an securityToken (this goes ok!)
post the securityToken + username + password to get logged in.
Public Class Form1
Const ConnectURL = "https:!!member.company.com/homepage/index"
Const LoginURL = "https:!!member.company.com/account/logn"
Private Function RegularPage(ByVal URL As String, ByVal CookieJar As CookieContainer) As String
Dim reader As StreamReader
Dim Request As HttpWebRequest = HttpWebRequest.Create(URL)
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
Request.AllowAutoRedirect = False
Request.CookieContainer = CookieJar
Dim Response As HttpWebResponse = Request.GetResponse()
reader = New StreamReader(Response.GetResponseStream())
Return reader.ReadToEnd()
reader.Close()
Response.Close()
End Function
Private Function LogonPage(ByVal URL As String, ByRef CookieJar As CookieContainer, ByVal PostData As String) As String
Dim reader As StreamReader
Dim Request As HttpWebRequest = HttpWebRequest.Create(URL)
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
Request.CookieContainer = CookieJar
Request.AllowAutoRedirect = False
Request.ContentType = "application/x-www-form-urlencoded"
Request.Method = "POST"
Request.ContentLength = PostData.Length
Dim requestStream As Stream = Request.GetRequestStream()
Dim postBytes As Byte() = Encoding.ASCII.GetBytes(PostData)
requestStream.Write(postBytes, 0, postBytes.Length)
requestStream.Close()
Dim Response As HttpWebResponse = Request.GetResponse()
For Each tempCookie In Response.Cookies
CookieJar.Add(tempCookie)
Next
reader = New StreamReader(Response.GetResponseStream())
Return reader.ReadToEnd()
reader.Close()
Response.Close()
End Function
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim CookieJar As New CookieContainer
Dim PostData As String
Try
Dim homePage As String = (RegularPage(ConnectURL, CookieJar))
Dim securityToken = homePage.Substring(homePage.IndexOf("securityToken") + 22, 36) 'momenteel 36 characters lang
PostData = "securityToken=" + securityToken + "&accountId=123456789&password=mypassword"
MsgBox(PostData)
Dim accountPage As String = (LogonPage(LoginURL, CookieJar, PostData))
Catch ex As Exception
MsgBox(ex.Message.ToString)
End Try
End Sub
End Class
This line causes the connection to be closed:
Dim requestStream As Stream = Request.GetRequestStream()
Is it possible that this company doesnt like the automated login and somehow notices that a application is used for logging in? How can I debug this? Fiddler doesn't seem to work. Is my only option WireShark as this seems kind of difficult to me.
Also is it weird that the connection is already is closed before I do the Post?
Are there other languages I can program this "easily" to rule out it's VB.net / .NET problem?
Have you attempted to capture the request using something like your browser's networking tools?
The auth process may have changed. Could even be some name or post data changes.
I got this fixed by:
double checking all the headers to be sent when using a browser
made sure all those headers where sent by the VB.NET application.
Not sure which one did the trick, but just always make sure you replicate all the headers that the browser would sent!
I have a HTTP class that gets content from URL's, POST's content to URL's etc and then returns the raw HTML content.
In the function inside of the class it detects if there is a HTTP error and if so I would like to return false but will this work if I have declared the function to return a String?
Code Sample of what I am trying to do (Note the Return Content & Return False if a HTTP error code is detected)
Public Function Get_URL(ByVal URL As String) As String
Dim Content As String = Nothing
Try
Dim request As Net.HttpWebRequest = Net.WebRequest.Create(URL)
' Request Settings
request.Method = "GET"
request.KeepAlive = True
request.AllowAutoRedirect = True
request.Timeout = MaxTimeout
request.CookieContainer = cookies
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.60 Safari/534.24"
request.Timeout = 60000
request.AllowAutoRedirect = True
Dim response As Net.HttpWebResponse = request.GetResponse()
If response.StatusCode = Net.HttpStatusCode.OK Then
Dim responseStream As IO.StreamReader = New IO.StreamReader(response.GetResponseStream())
Content = responseStream.ReadToEnd()
End If
response.Close()
Catch e As Exception
HTTPError = e.Message
Return False
End Try
Return Content
End Function
And usage example:
Dim Content As String = Get_URL("http://www.google.com/")
If Content = False Then
MessageBox.Show("A HTTP Error Occured: " & MyBase.HTTPError)
Exit Sub
End If
Usually in this type of scenario, you would throw a new exception with more detailed information, and let the exception bubble up to the processed by the main code (or just let the original exception bubble up without Catching it in the first place).
Catch e As Exception
' wrap the exception with more info as a nested exception
Throw New Exception("Error occurred while reading '" + URL + "': " + e.Message, e)
End Try
Inside the usage example:
Dim content As String = ""
Try
content = Get_URL("http://www.google.com/")
Catch e As Exception
MessageBox.Show(e.Message)
Exit Sub
End Try