I tried to get the webpage source of pages under www.digikey.com
It was working fine long back but now the website is rejecting the request, but it shows no problem in web browser , issue happens when trying to access via code and when i examine the page sorce received now its written
There was a problem with your request.
We are unable to process your request.
Please return to the previous page to try again or contact Digi-Key Webmaster if you feel that you have received this message in error. Please reference the following incident number so we may assist you with this error.
The code i used was:
Dim PartURL As String
PartURL = "http://www.digikey.com"
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(PartURL)
Dim response As System.Net.HttpWebResponse = request.GetResponse()
Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream())
Dim sourcecode As String = sr.ReadToEnd()
when i changed the url to google.com its working well.
The website www.digikey.com works without a problem in web browsers. It shows error only when trying to access via the code. So i thought it may have something in relation with the code. Is it because digikey is rejecting the request. Is there any way i can get the source code of pages under www.digikey.com
Try this:
Dim PartURL As String
PartURL = "http://www.digikey.com"
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(PartURL)
request.UserAgent = "Mozilla/5.0 (Windows NT 5.1; rv:2.0b8) Gecko/20100101 Firefox/4.0b8"
request.Accept = "Accept: text/html,application/xhtml+xml,application/xml"
Dim response As System.Net.HttpWebResponse = request.GetResponse()
Dim sr As System.IO.StreamReader = New System.IO.StreamReader(response.GetResponseStream())
Dim sourcecode As String = sr.ReadToEnd()
Related
I get
'underlying connection was closed'
when running the code below. I am using vb.net 2012 (I must use this version) with the RestSharp library and am trying to retrieve product data from a bigcommerce.com store. This is a simple vb.net 2012 console program that once I get working I can build upon. I have tried changing around the code somewhat even making certain things redundant like the method and URL but I can't get it to work.
Dim client As New RestClient
client.BaseUrl = New Uri("https://api.bigcommerce.com/stores/mystorehash/v3/catalog/products")
Dim request As New RestRequest("https://api.bigcommerce.com/stores/mystorehash/v3/catalog/products", Method.GET)
request.AddHeader("Accept", "application/json")
request.AddHeader("Content-Type", "application/json")
request.AddHeader("X-Auth-Client", "notactualvaluenotactualvalue")
request.AddHeader("X-Auth-Token", "notactualvaluenotactualvalue")
request.Method = Method.GET
Dim response As New RestResponse
response = client.ExecuteAsGet(request, Method.GET)
Console.WriteLine("response.Content=" & response.Content)
Console.WriteLine("response.ErrorMessage=" & response.ErrorMessage)
Console.WriteLine("response.ResponseStatus=" & response.ResponseStatus)
Console.WriteLine("response.IsSuccessful=" & response.IsSuccessful)
Console.WriteLine("response.Headers.Count=" & response.Headers.Count)
Output:
Any help would be appreciated, hopefully I'm doing something stupid that can be easily fixed
For anyone else that finds this the full implementation of Nathan Booker's suggestion is below. When you attempt to access the Big Commerce api in a VB.NET application you need to specify TLS 1.2. If you don't you'll recieve an HTTP status of 502 - System.IO.IOException Authentication failed because the remote party has closed the transport stream. The solve is done like:
ServicePointManager.Expect100Continue = True
ServicePointManager.SecurityProtocol = CType(3072, SecurityProtocolType)
ServicePointManager.DefaultConnectionLimit = 9999
Dim request As HttpWebRequest = CType(WebRequest.Create("https://api.bigcommerce.com/stores/<Redacted>/v3/catalog/products"), HttpWebRequest)
request.AllowAutoRedirect = True
request.ContentType = "application/json"
request.Accept = "application/json"
request.Method = "GET"
request.Headers.Add("X-Auth-Client", "<Redacted>")
request.Headers.Add("X-Auth-Token", "<Redacted>")
Dim response As WebResponse = request.GetResponse()
Diagnostics.Debug.WriteLine((CType(response, HttpWebResponse)).StatusDescription)
Dim dataStream As Stream = response.GetResponseStream()
Dim reader As StreamReader = New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
Diagnostics.Debug.WriteLine(responseFromServer)
reader.Close()
response.Close()
This may be related to your HTTP version or (more likely) SSL/TLS protocol.
If possible, please make sure you're using HTTP 1.1 and TLS 1.2.
I'm trying to submit a POST request containing JSON to a given endpoint. However the below code sends me to a login page which is the default functionality for all non-/api endpoints. Because Postman doesn't direct me to the login page but correctly hits the API, I'm assuming the problem is in this C# code.
Dim RemoteUrl As String = "https://my-site.com/api/tournament/results"
Dim result As String = ""
Dim xmlHttpReq As HttpWebRequest = CType(WebRequest.Create(RemoteUrl), HttpWebRequest)
xmlHttpReq.Method = "POST"
xmlHttpReq.ContentType = "application/json"
With (New StreamWriter(xmlHttpReq.GetRequestStream))
.Write(Json)
.Flush()
.Close()
Dim httpResponse As HttpWebResponse = CType(xmlHttpReq.GetResponse(), HttpWebResponse)
With (New StreamReader(httpResponse.GetResponseStream))
result = .ReadToEnd
End With
End With
Any idea what steps I need to take to debug this?
im currently trying to login to my website, this works however it wont redirect when it logs in. here is my code
Dim request As WebRequest = WebRequest.Create("http://www.jamiehayles.com/test/login.php")
request.Method = "POST"
Dim postData As String = "username=testing&password=lmao1234"
Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postData)
request.ContentType = "application/x-www-form-urlencoded"
request.ContentLength = byteArray.Length
Dim dataStream As Stream = request.GetRequestStream()
dataStream.Write(byteArray, 0, byteArray.Length)
dataStream.Close()
Dim response As WebResponse = request.GetResponse()
Console.WriteLine(CType(response, HttpWebResponse).StatusDescription)
dataStream = response.GetResponseStream()
Dim reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
RichTextBox1.Text = responseFromServer
reader.Close()
dataStream.Close()
response.Close()
my question is, do i need cookies for it to redirect? and if so any help guys?
Take a look at what you're doing with the response from the server:
Dim responseFromServer As String = reader.ReadToEnd()
RichTextBox1.Text = responseFromServer
All you're doing is reading it and placing the content of the response into a text box. You're not actually responding to a redirect in any way.
A "redirect" means that the server responds with a specific status code and header in the response. The status code tells the browser that the response is an instruction to redirect to another resource. The header contains the new resource that the browser should request.
A web browser checks the response for this code/header and performs a new request. Your code doesn't. It just shows the response to the user, regardless of what it is.
So, to illustrate, a normal redirect happens like this:
Browser: I'm requesting this resource.
Server: Ok, I've received your request, but the resource you really want is actually over there.
Browser: Ok, then I'm now requesting that resource over there.
Server: Ok, here you go.
Your code, however, is more like this:
Code: I'm requesting this resource.
Server: Ok, I've received your request, but the resource you really want is actually over there.
Code: [no further action is taken]
The server instructed your code, in an HTTP-standard way, to request a new resource. Your code simply ignored that instruction.
You need to check the status code of the response, as well as the headers of the response, and make new requests accordingly.
I'm trying to to make a new post through my vb.net application using blogger api.
But I'm fail every time.
Sometimes it's return 403 forbidden error some times Unauthorised error.
Please Help.
Dim mBlogID As String = "5861877551002158183"
Dim AuthToken As String = "AIza......xxxx..........E6g"
Dim post As String = "{""kind"": ""blogger#post"", ""blog"": { ""id"": """ & mBlogID & """}, ""title"": ""abc-title"", ""content"": ""abc-cont""}"
Dim request As HttpWebRequest = DirectCast(WebRequest.Create("https://www.googleapis.com/blogger/v3/blogs/" & mBlogID & "/posts?key=" & AuthToken), HttpWebRequest)
request.Method = "POST"
request.ContentLength = post.Length
request.ContentType = "application/json"
request.Headers.Add("Authorization: ", AuthToken) '<--- error here
Using requestStream As Stream = request.GetRequestStream()
Dim postBuffer As Byte() = Encoding.ASCII.GetBytes(post)
requestStream.Write(postBuffer, 0, postBuffer.Length)
End Using
Using response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse) '<--- Unauthorised error or 403 error here
Using responseStream As Stream = response.GetResponseStream()
Using responseReader As New StreamReader(responseStream)
'Dim json As String = responseReader.ReadToEnd()
'Dim PostURL As String = Regex.Match(json, """url"": ?""(?<id>.+)""").Groups("id").Value
MsgBox(json) 'want to read json response here.
'MsgBox(PostURL)
End Using
End Using
End Using
This code return this error :Specified value has invalid HTTP Header characters.
Parameter name: name
can anybody fix it? I just want to make a new post to blogger and read its URL.
Project Information:-
Platform: Visual Basic 2010
Blogger API Version: V3
I Suggest view this solution. The post has fully explanations.
I need to use HTTPWebRequest to login to an external website and redirect me to the default page. My code below is behind a button - when clicked it currently tries to do some processing but stays on the same page. I need it to redirect me to the default page of the external website without seeing the login page. Any help on what I'm doing wrong?
Dim loginURL As String = "https://www.example.com/login.aspx"
Dim cookies As CookieContainer = New CookieContainer
Dim myRequest As HttpWebRequest = CType(WebRequest.Create(loginURL), HttpWebRequest)
myRequest.CookieContainer = cookies
myRequest.AllowAutoRedirect = True
myRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1"
Dim myResponse As HttpWebResponse = CType(myRequest.GetResponse(), HttpWebResponse)
Dim responseReader As StreamReader
responseReader = New StreamReader(myResponse.GetResponseStream())
Dim responseData As String = responseReader.ReadToEnd()
responseReader.Close()
'call a function to extract the viewstate needed to login
Dim ViewState As String = ExtractViewState(responseData)
Dim postData As String = String.Format("__VIEWSTATE={0}&txtUsername={1}&txtPassword={2}&btnLogin.x=27&btnLogin.y=9", ViewState, "username", "password")
Dim encoding As UTF8Encoding = New UTF8Encoding()
Dim data As Byte() = encoding.GetBytes(postData)
'POST to login page
Dim postRequest As HttpWebRequest = CType(WebRequest.Create(loginURL), HttpWebRequest)
postRequest.Method = "POST"
postRequest.AllowAutoRedirect = True
postRequest.ContentLength = data.Length
postRequest.CookieContainer = cookies
postRequest.ContentType = "application/x-www-form-urlencoded"
postRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20100101 Firefox/13.0.1"
Dim newStream = postRequest.GetRequestStream()
newStream.Write(data, 0, data.Length)
newStream.Close()
Dim postResponse As HttpWebResponse = CType(postRequest.GetResponse(), HttpWebResponse)
'using GET request on default page
Dim getRequest As HttpWebRequest = CType(WebRequest.Create("https://www.example.com/default.aspx"), HttpWebRequest)
getRequest.CookieContainer = cookies
getRequest.AllowAutoRedirect = True
Dim getResponse As HttpWebResponse = CType(getRequest.GetResponse(), HttpWebResponse)
'returns statuscode = 200
FYI - when i add in this code at the end, i get the HTML of the default page I'm trying to redirect to
Dim responseReader1 As StreamReader
responseReader1 = New StreamReader(getRequest.GetResponse().GetResponseStream())
responseData = responseReader1.ReadToEnd()
responseReader1.Close()
Response.Write(responseData)
Any help on whats missing to get the redirect working?
Cheers
The HttpWebRequest only automatically redirects you if the server sends an HTTP 3xx redirection status with a Location field in the response. Otherwise you are supposed to manually navigate to the page by using Response.Redirect, for example. Also keep in mind that the automatic redirection IGNORES ANY COOKIES sent by the server. That may be the problem in your case if the server is actually sending a redirection status.