It's like 5h I'm testing backconnect proxy I have which rotate on each request, but when I try to connect with proxy credentials (username , pass) it always connect and receive response from stream even if I type wrong pass, which makes me sick.
My code:
Public Shared Sub TestProxyConn()
Try
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12
Dim _proxyIP As String = Form1.txtRotatingIP.Text
Dim _proxyPort As Integer = Convert.ToInt32(Form1.txtRotatingPort.Text)
Dim _proxyUser As String = Form1.txtRotatingUser.Text
Dim _proxyPass As String = Form1.txtRotatingPass.Text
Dim myProxy As IWebProxy = New WebProxy("http://" & _proxyIP & ":" & _proxyPort)
Dim _url As String = "https://www.whatsmyua.info/"
Dim the_request As HttpWebRequest = WebRequest.Create(_url)
If Form1.chAuthMethod.Checked = True Then
the_request.UseDefaultCredentials = False
myProxy.Credentials = New NetworkCredential(_proxyUser, _proxyPass)
End If
the_request.Proxy = myProxy
the_request.Timeout = 10000
the_request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
Dim the_response As System.Net.HttpWebResponse = the_request.GetResponse
Dim stream_reader As New System.IO.StreamReader(the_response.GetResponseStream())
Dim _source As String = stream_reader.ReadToEnd
If _source.Contains("my user agent") Then
MessageBox.Show("Connected!")
End If
Catch ex As System.Net.WebException
MessageBox.Show("Connection failed!")
End Try
End Sub
Related
I want to escape the \/\/ from url to get thumb with Uri.UnescapeDataString("\/\/") on this url from the HTML source using request:
https:\/\/scontent.fhel5-1.fna.fbcdn.net\/v\/t15.5256-10\/fr\/cp0\/e15\/q65\/48734886_2061090874011522_801576553975644160_n.jpg?_nc_cat=107&_nc_sid=08861d&_nc_ohc=4jeujL2uPDwAX_FBn97&_nc_ht=scontent.fhel5-1.fna&oh=672481858f1d7a2bd2655da89b57bacd&oe=5F63E18C
So my thumb will be decoded and those sign replaced so it can be parsed as string in show my the picturebox imgCover.ImageLocation = _thumbURL.ToString()
My code:
Private Sub getNFO()
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
Dim apiURL As String = "https://m.facebook.com/watch/?v="
Dim _videoID As String = txtURL.Text
Try
Dim the_request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(apiURL + _videoID)
the_request.UserAgent = "Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1"
the_request.Timeout = 5000
Dim the_response As System.Net.HttpWebResponse = the_request.GetResponse
Dim stream_reader As System.IO.StreamReader = New System.IO.StreamReader(the_response.GetResponseStream())
Dim yt_source As String = stream_reader.ReadToEnd
If String.IsNullOrEmpty(yt_source) Then
MessageBox.Show("Facebook ID Request Error")
Else
Dim r1 As Regex = New Regex("(?<=description"":"").*?([^[]+?)(?="")")
Dim m1 = r1.Match(yt_source)
_titleURL = m1.Value
Dim r2 As Regex = New Regex("(?<=thumbnailUrl"":"").*?([^[]+?)(?="")")
Dim m2 = r2.Match(yt_source)
_thumbURL = m2.Value
End If
lbl_Title.Text = _titleURL.ToString()
imgCover.ImageLocation = _thumbURL.ToString()
Catch ex As Exception
MessageBox.Show(ex.Message)
End Try
End Sub
Finally make it bymyself :)
Dim decodedThumbURL = _thumbURL.Replace("\", String.Empty)
so i am trying to login to a website using httpwebrequest. the post data i got from a http debugger is
code i am trying is:
Dim postData As String = "securitycheck=85b39cc89f04bc1612ce9d0c384b39ca&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=jawademail&pass=jawadpass"
Dim tempCookies As New CookieContainer
Dim encoding As New UTF8Encoding
Dim byteData As Byte() = encoding.GetBytes(postData)
Dim postReq As HttpWebRequest = DirectCast(WebRequest.Create("https://www.dreamstime.com/securelogin.php"), HttpWebRequest)
postReq.Method = "POST"
postReq.KeepAlive = True
postReq.CookieContainer = tempCookies
postReq.ContentType = "application/x-www-form-urlencoded"
postReq.Referer = "https://www.dreamstime.com/login.php"
postReq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)"
postReq.ContentLength = byteData.Length
Dim postreqstream As Stream = postReq.GetRequestStream()
postreqstream.Write(byteData, 0, byteData.Length)
postreqstream.Close()
Dim postresponse As HttpWebResponse
postresponse = DirectCast(postReq.GetResponse(), HttpWebResponse)
tempCookies.Add(postresponse.Cookies)
logincookie = tempCookies
Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
Dim thepage As String = postreqreader.ReadToEnd
RichTextBox1.Text = thepage
thsi code does not seem to post data in website i get referer page code in richtextbox after running the code.
First GET the page, find the "securitycheck" in its source and extract it.
Combine it with the rest of your data then send it with POST.
Ok so I felt like trying:
Dim LoginData As String
Dim LoginCookies As New CookieContainer() 'Move this outside of sub/function so you can use it later
Dim LoginRequest As HttpWebRequest = WebRequest.Create("https://www.dreamstime.com/login.php")
LoginRequest.CookieContainer = LoginCookies
LoginRequest.KeepAlive = True
LoginRequest.AllowAutoRedirect = True
LoginRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0"
Dim LoginResponse As HttpWebResponse = LoginRequest.GetResponse()
Dim LoginResponseRead As StreamReader = New StreamReader(LoginResponse.GetResponseStream())
Using LoginResponseRead
Do
Dim line As String = LoginResponseRead.ReadLine
If line.Contains("var securitycheck=") Then
LoginData = "securitycheck=" & line.Substring(line.IndexOf("=") + 2, line.LastIndexOf("'") - line.IndexOf("=") - 2)
Exit Do
End If
Loop
End Using
Dim byteData As Byte() = Encoding.UTF8.GetBytes(LoginData)
LoginRequest = WebRequest.Create("https://www.dreamstime.com/securelogin.php")
LoginRequest.CookieContainer = LoginCookies
LoginRequest.Method = "POST"
LoginRequest.KeepAlive = True
LoginRequest.ContentType = "application/x-www-form-urlencoded"
LoginRequest.Referer = "https://www.dreamstime.com/login.php"
LoginRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)"
LoginRequest.ContentLength = byteData.Length
Dim postreqstream As Stream = LoginRequest.GetRequestStream()
postreqstream.Write(byteData, 0, byteData.Length)
postreqstream.Close()
LoginResponse = LoginRequest.GetResponse()
LoginResponseRead = New StreamReader(LoginResponse.GetResponseStream())
Dim thepage As String = LoginResponseRead.ReadToEnd
'Now with GET request grab whatever you want, DON'T forget to use cookie.
Result
>>>securitycheck=183d5abdb01f288aacbe5b2893555ec5
Dim email As String = "something"
Dim password As String = "somethingelse"
LoginData &= "&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=" & email & "&pass=" & password
>>>securitycheck=183d5abdb01f288aacbe5b2893555ec5&do_action=log_into_system&jump_to=https%3A%2F%2Fwww.dreamstime.com%2F&uname=something&pass=somethingelse
There, practically done.
I took a look at this myself and it appears that with every login request a token is sent that identifies your "session", specifically:
securitycheck=85b39cc89f04bc1612ce9d0c384b39ca
This token changes every time you login, and if it isn't valid the site redirects you back to the login page, asking you to login again.
Sites usually do this to prevent Cross-Site Request Forgery (CSRF). This means that you will most likely not be able to login to this site without using an actual web browser.
Here's the code that was tested and it works. It uses System.Net.Http.HttpClient rather WebClient (since it supports concurrent requests). This code is just a model since its main goal is to show the idea how to work with this site. There are additional explanations in comments. You also need to import System.Web dll.
Imports System.Net.Http
Imports System.Web
Imports System.Text.RegularExpressions
Public Class TestForm
Private Const URL_MAIN$ = "https://www.dreamstime.com"
Private Const URL_LOGIN$ = "https://www.dreamstime.com/securelogin.php"
Private Const URL_LOGOUT$ = "https://www.dreamstime.com/logout.php "
Private Const USER_AGENT$ = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) " +
"AppleWebKit/537.36 (KHTML, Like Gecko) " +
"Chrome/68.0.3440.15 Safari/537.36 OPR/55.0.2991.0 " +
"(Edition developer)"
Private Const LOGIN$ = "<USER_NAME>"
Private Const PASS$ = "<USER_PASSWORD>"
Private token$
Private Async Sub OnGo() Handles btnGo.Click
Dim html$
Using client = New HttpClient()
client.DefaultRequestHeaders.Add("User-Agent", USER_AGENT)
Using req = New HttpRequestMessage(HttpMethod.Get, URL_MAIN)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Search for security token
Dim m = Regex.Match(
html,
"<input type=""hidden"" name=""securitycheck"" value=""(?'token'\w+)"">")
If Not m.Success Then
MessageBox.Show("Could not find security token.")
Return
End If
'// Get security token
token = m.Groups("token").Value
'// Try to login.
'// For logging to work, we need to use FormUrlEncodedContent class.
'// Also we need to use it every time we do POST requests.
'// No need for it for GET requests (as long as the HttpClient is the same).
Using req = New HttpRequestMessage(HttpMethod.Post, URL_LOGIN) With
{
.Content = GetFormData()
}
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Go to main page to check we're logged in.
'// "html" variable now MUST contain user's account name.
Using req = New HttpRequestMessage(HttpMethod.Get, URL_MAIN$)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
'// Logout.
'// "html" variable now MUST NOT contain user's account name.
Using req = New HttpRequestMessage(HttpMethod.Get, URL_LOGOUT)
Using resp = Await client.SendAsync(req)
html = Await resp.Content.ReadAsStringAsync()
End Using
End Using
End Using
End Sub
Function GetFormData() As FormUrlEncodedContent
Return New FormUrlEncodedContent(New Dictionary(Of String, String) From
{
{"securitycheck", token},
{"do_action", "log_into_system"},
{"jump_to", ""},
{"uname", HttpUtility.HtmlEncode(LOGIN)},
{"pass", HttpUtility.HtmlEncode(PASS)}
})
End Function
End Class
I want to take the Flight schedule from the following address:
http://fo-apac.ttinteractive.com/Zenith/FrontOffice/(S(nves1yv4xxoia40cmotixof1))/USBangla/en-GB/BookingEngine/SearchFlights?__cnv=tShqK&json={"BookingPathArguments":null,"OriginDestinations":[{"IsOpen":false,"DataIdOrigin":6337,"DataIdDestination":6707,"DateTime":"2016-04-27T00:00:00.000"}],"TravelerTypes":[{"DataId":1,"TravelerCount":1},{"DataId":2,"TravelerCount":0},{"DataId":3,"TravelerCount":0}],"Currency":{"Code":"BDT"},"PromoCode":null,"DisplayRealAvailability":false,"Visibility":0,"ExtendedSearchDayCount":3}
If you paste the address to browser address bar and use firebug (or any fiddler), you will see this page sends 3 jquery ajax calls to bring the schedule. The following ajax POST actually fetch the schedule.
http://fo-apac.ttinteractive.com/Zenith/FrontOffice/(S(nves1yv4xxoia40cmotixof1))/USBangla/en-GB/FlexibleFlightStaticAjax/FlexibleFlightListLoadSelectedDays?__cnv=mxw0s
PostData : SaleConditionAccepted=false&ExtendedSearchDayCount=3&DoNotCheckCache=false&AlreadyLoggedIn=false&TempDataGuid=nves1yv4xxoia40cmotixof1&CurrencyCode=BDT&FareBasisDataId=&Travelers[0][DataId]=1&Travelers[0][Count]=1&UserSelections[0][SelectedDate]=2016-04-27T00:00:00&UserSelections[0][ReferenceDate]=2016-04-27T00:00:00&UserSelections[0][DataIdOrigin]=6337&UserSelections[0][DataIdDestination]=6707&UserSelections[0][GenericClassDataId]=&UserSelections[0][SelectedSegments]=&JsonPrepareBookingRequest=&PromoCode=
I am sending the request with httpwebrequest, but for unknown reason, I miss the session. I used CookieContainer to keep the cookies. I used the following function to send httprequest:
Public Function GetPostWP(ByVal Url As String, ByVal CkCont As CookieContainer, Optional ByVal PostData As String = "", Optional ByVal refSite As String = "") As String
Dim pStr As String = ""
Try
Dim Http As HttpWebRequest = WebRequest.Create(Url)
If refSite <> "" Then Http.Referer = refSite
Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
Http.CookieContainer = CkCont 'Initial CkCont is Nothing
Http.KeepAlive = True
Http.AllowAutoRedirect = True
'Http.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6"
Http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
Http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36"
If PostData <> "" Then
Http.Method = "POST"
Http.ContentLength = PostData.Length
Http.ContentType = "application/x-www-form-urlencoded"
Dim PostStream As StreamWriter = New StreamWriter(Http.GetRequestStream())
PostStream.Write(PostData)
PostStream.Close()
End If
Using WebResponse As HttpWebResponse = Http.GetResponse()
Dim responseStream As Stream = WebResponse.GetResponseStream()
If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If
Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
pStr = reader.ReadToEnd()
responseStream.Close()
End Using
tmpCky = CkCont 'tmpCky is a Public CookieContainer Variable to hold cookies for future use.
GetPostWP = pStr
Catch ex As Exception
GetPostWP = "Error : " & ex.Message
End Try
End Function
Dim Cky As New CookieContainer
Dim Txt as String = GetPostWP(PostAddress, Cky, PostData, RefAdd)
Cky = tmpCky
Can anyone analyze the page, please?
I'm trying to connect to a third party website, and then get the source code.
It worked well for me, and after some time I tried sign in again and then i got an error message:
"{" The server committed a protocol violation. Section = ResponseStatusLine "}.
After quick search on google i found out that i sholud to add to "app.config" the following entry: <httpWebRequest useUnsafeHeaderParsing = true/>
After that it worked fine. But, I get the source code as a guset, and not as "connected User".
I tried another site that does not require "UseUnsafeHeaderParsing = true" and it worked well.
It looks like "UseUnsafeHeaderParsing = true" disruptive cookies?
*Sorry for my english, this is not my native language.
this is my code:
Private siteCookies As New Net.CookieContainer()
Function siteRequest(url As String, ByVal Method As String, Optional ByVal data As String = Nothing) As String
Static soucrecode As String = Nothing
Const UserAgent As String = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
Try
Dim Request As Net.HttpWebRequest = Net.WebRequest.Create(url)
Request.Accept = "*/*"
Request.Timeout = 10000
Request.Method = Method
Request.UserAgent = UserAgent
Request.AllowAutoRedirect = True
Request.CookieContainer = siteCookies
If Request.Method = "POST" AndAlso data IsNot Nothing Then
Dim postBytes() As Byte = New UTF8Encoding().GetBytes(data)
Request.ContentType = "application/x-www-form-urlencoded"
Request.ContentLength = postBytes.Length
Request.GetRequestStream().Write(postBytes, 0, postBytes.Length)
End If
Dim Response As Net.HttpWebResponse = Request.GetResponse()
soucrecode = New IO.StreamReader(Response.GetResponseStream).ReadToEnd()
Response.Close()
Catch e As Exception
Console.WriteLine(e.Message)
End Try
Return soucrecode
End Function
using:
txtLoginSoucre.Text = siteRequest("http://www.SomeSite.com/login.php?do=login", "Post", "username=myUser&password=MyPass") ' Login to site
txtSoucre.Text siteRequest("http: //www.SomeSite.com", "Get") 'Grab soucrecode
Just add
Request.KeepAlive = False
iam trying to crawl some sites. It works like a charm. But there is a major problem. On some pages (not mutch) I'm getting some weird characters instead of html code.
It looks like this:
;�<cS���u�/�qYa$�4l7�.�Q�7&��O����� Z�D}z��/���� ��u����V���lWY|�n5�1�We����GB�U��g{�� �|Ϸ����*�Q��0���nb�o�߯�����[b��/����#CƑ����D{{/n��X�!� �Et�X"����?��˩����8\y��&
If I'll open it in my browser, there is no Problem at all.
I dont understand why.
My HTTP Header says:
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding:gzip,deflate,sdch
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:max-age=0 Connection:keep-alive User-Agent:Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/29.0.1547.66 Safari/537.36
I think it has something to do with the Accept request.Accept = "*/*"
Thats my webrequest:
Public Class Http
Dim cookieCon As New CookieContainer
Dim request As HttpWebRequest
Dim response As HttpWebResponse
Public Function GetRequest(ByVal Params() As Object)
Dim url As String = Params(0)
Dim mycookie As String = Params(1)
'request.AllowAutoRedirect = True
request = CType(HttpWebRequest.Create(url), HttpWebRequest)
request.CookieContainer = New CookieContainer()
request.Method = "GET"
request.Timeout = 20000
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36"
'request.ContentType = "application/x-www-form-urlencoded"
request.Accept = "*/*"
If Not mycookie Like "nocookie" Then
request.Headers("Cookie") = mycookie
End If
response = CType(request.GetResponse(), HttpWebResponse)
Dim html(1) As String
html(0) = request.Address.ToString()
html(1) = New StreamReader(response.GetResponseStream()).ReadToEnd()
Return html
End Function
Thanks.
The data you are downloading is GZip compressed. You need to decompress it. Change your function to this:
Dim request As HttpWebRequest
Dim response As HttpWebResponse
Public Function GetRequest(ByVal Params() As Object) As String()
Dim url As String = Params(0)
Dim mycookie As String = Params(1)
'request.AllowAutoRedirect = True
request = CType(HttpWebRequest.Create(url), HttpWebRequest)
request.CookieContainer = New CookieContainer()
If Not mycookie Like "nocookie" Then
request.Headers("Cookie") = mycookie
End If
request.AutomaticDecompression = DecompressionMethods.GZip
response = CType(request.GetResponse(), HttpWebResponse)
Dim html(1) As String
html(0) = request.Address.ToString()
html(1) = New StreamReader(response.GetResponseStream).ReadToEnd()
Return html
End Function
Usage:
Dim params(1) As Object
params(0) = url
Dim page As String = GetRequest(params)(1)