Below is the example code I'm using to get this to work and it does work if I try to read yahoo.com.
Here is the problem. The address I need to read is a java servlet that processes parameters passed in, generates a text document on the server, and then redirects to another URL and returns the address of the text file on the server. I then need to download that text file and process it. I'm having problems connecting to the first URL with the parameters and I think it has to do with the redirect.
I'm using the WebRequest object and I've tried using the HttpWebRequest object. Are there any other objects that support redirects?
TIA
Dim reader As StreamReader
Dim request As WebRequest
Dim response As WebResponse
Dim data As String = ""
Try
request = WebRequest.Create("URL Here")
request.Timeout = 30000
response = request.GetResponse()
reader = New StreamReader(response.GetResponseStream())
data = reader.ReadToEnd()
Catch ex As Exception
MsgBox(ex.Message)
End Try
Return data
Edit
I just tested out HttpWebRequest.Create() and that does handle the 301 and 302 fine with out extra code.
Can you post the error you are seeing
You could cast the WebResponse to a HttpWebResponse:
I need to convert this to VB... but it might help you start:
var response = request.GetResponse() as HttpWebResponse;
if (response.StatusCode == HttpStatusCode.Moved || response.StatusCode == HttpStatusCode.Redirect)
{
// Follow Redirect, new request based off Redirect
}
// Read Data
I believe you just have to set the AutoRedirect property.
request.AutoRedirect = true;
webRequest = webRequest.Create(URL)
webresponse = webRequest.GetResponse()
inStream = New StreamReader(webresponse.GetResponseStream())
Read URL full source code
winston
I think I found something that will work.
I used a WebBrowser control instead.
Have a button that runs this code...
WebBrowser1.Navigate("URL Here")
And this function to process once the request returns.
Private Sub WebBrowser1_Navigated(ByVal sender As System.Object, ByVal e As System.Windows.Forms.WebBrowserNavigatedEventArgs) Handles WebBrowser1.Navigated
MsgBox(WebBrowser1.DocumentText)
End Sub
Related
I am trying to login to one web page via my Windows Form application (VB 2010) and get some response, but I don't know how to do it.
The server service is described here:
https://api.developer.betfair.com/services/webapps/docs/display/1smk3cen4v3lu3yomq5qye0ni/Interactive+Login+from+a+Desktop+Application
On beginning I tried to insert WebBrowser Control to my form, but I read many articles, also here, that using BeforeNavigate2 event is an old way.
I would like to do this:
Send HTTP request to webserver including username and password
Catch the post data sent to the redirect URL
Read the POST request body an get loginStatus and productToken (SSOID)
This is my code:
Private Sub getPOST()
' Create a request for the URL
Dim request As WebRequest = _
WebRequest.Create("https://identitysso.betfair.com/view/login?product=82&url=https://www.betfair.com&username=abc&password=abc")
request.Method = "POST"
' Get the response
Dim response As HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
' Get the stream containing content returned by the server.
Dim dataStream As Stream = response.GetResponseStream()
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
Dim responseFromServer As String = reader.ReadToEnd()
' Display the content.
RichTextBox1.Text = responseFromServer
' Cleanup the streams and the response.
reader.Close()
dataStream.Close()
response.Close()
End Sub
But this code returns HTML code of page where are not loginStatus and SSOID.
I examined this page and discovered there are a few problems with your code:
First you are posting to the wrong URL because the www.betfair.com/view/login form has action="/api/login", not "/view/login". Second, you are passing your username/password in the QueryString rather than in POSTDATA... this form uses method=post. Third, you are not passing all required fields. This code snippet should remedy these described problems, as well as answer your question about obtaining the redirect URL:
Sub TestWebClient()
' Create a request for the URL
' You were using the wrong URL & passing the values improperly
' I also changed this to HttpWebRequest, used to be WebRequest.
Dim request As HttpWebRequest = WebRequest.Create("https://identitysso.betfair.com/api/login") '
request.Method = "POST"
' New code added
request.ContentType = "application/x-www-form-urlencoded"
' New code added, this prevents the following of the 302 redirect in the Location header
request.AllowAutoRedirect = False
' New code added, this sends the values as POSTDATA, which is contained inside the HTTP POST request rather than in the URL
Using writer As New StreamWriter(request.GetRequestStream)
writer.Write("username=abc")
writer.Write("&password=abc")
writer.Write("&login=true")
writer.Write("&redirectMethod=POST")
writer.Write("&product=82")
writer.Write("&url=https://www.betfair.com/")
writer.Write("&ioBlackBox=")
End Using
' Get the response
' This will print the Location header (aka "Redirect URL") on the screen for you
' Make decisions as needed.
Dim response As HttpWebResponse = request.GetResponse()
Console.WriteLine("HTTP RESPONSE HEADERS:")
For Each item In response.Headers
Console.WriteLine(item & "=" & response.Headers(item))
Next
Console.ReadLine()
End Sub
I am making a Windows Phone app and I am trying to get JSON data from a URL. The user needs to be logged into the website (which hosts the JSON data) in order to get JSON data and I cannot use the Web Browser control to display the data and then extract the string since the browser doesn't recognize it (for some weird reason) and asks to search for an app on Store which can handle that JSON file type. (If I open the URL in desktop browser on my Windows PC, I can see the raw JSON data). I can't use normal HTTPWebRequest or WebClient as to get JSON data the login cookies needs to be set (the JSON data is user-specific) and I can't extract the cookies from Web browser control and use it with WebClient or HTTPWebRequest. So the best thing I can do is use a special internal instance of IWebRequestCreate that is used internally by the WebBrowser. By opening background HTTP requests with that class, the cookies get automatically set as if they were created/sent by the WebBrowser control. But my code is not returning the JSON data, I get blank response, as in the string resp is empty.
Below is the code:
Dim browser = New WebBrowser()
Dim brwhttp = GetType(WebRequestCreator).GetProperty("BrowserHttp")
Dim requestFactory = TryCast(brwhttp.GetValue(Browser, Nothing), IWebRequestCreate)
Dim uri = New Uri("http://api.quora.com/api/logged_in_user?fields=inbox,notifs,following,followers")
Dim req = requestFactory.Create(uri)
req.Method = "GET"
req.BeginGetResponse(New AsyncCallback(AddressOf request_Callback), req)
Private Sub request_Callback(asyncResult As IAsyncResult)
Dim webRequest As HttpWebRequest = DirectCast(asyncResult.AsyncState, HttpWebRequest)
Dim webResponse As HttpWebResponse = DirectCast(webRequest.EndGetResponse(asyncResult), HttpWebResponse)
Dim tempStream As New MemoryStream()
webResponse.GetResponseStream().CopyTo(tempStream)
Dim sr As New StreamReader(tempStream)
Dim resp As String = sr.ReadToEnd
End Sub
What's wrong?
I found that CopyTo can leave the Stream's pointer at the end of the buffer, you probably need to reset tempStream's pointer to the beginning before attempting to read it with the StreamReader, here's the code...
webResponse.GetResponseStream().CopyTo(tempStream);
tempStream.Seek(0, SeekOrigin.Begin);
Dim sr As New StreamReader(tempStream);
I am coding a whitelist for my application in VB.NET.
I am using the HttpWebRequest Method and HttpWebResponse.
If the Whitelist Host is down, the whitelist is bypassed and the program is available to anyone which is a vulnerability.
Public Function GetWhitelist(ByVal PageURL As String) As String
Dim S As String = ""
Try
Dim Request As HttpWebRequest = WebRequest.Create("WHITELIST URL HERE")
Dim Response As HttpWebResponse = Request.GetResponse()
Using Reader As StreamReader = New StreamReader(Response.GetResponseStream())
S = Reader.ReadToEnd
End Using
Catch ex As Exception
Debug.WriteLine("Start Program Error. Handle:0")
End Try
Return S
End Function
I want to give the user an error if the website is down, any ideas?
Regards.
What is your error condition? This function alone doesn't prevent anything from happening in the event of an exception. The function still exits with returning a string. So any consuming code would have to look for error conditions and handle them accordingly.
If the request fails, you should meaningfully handle the exception. Two ideas off the top of my head would include:
1) Let the exception bubble up the stack:
Public Function GetWhitelist(ByVal PageURL As String) As String
Dim S As String = ""
Dim Request As HttpWebRequest = WebRequest.Create("WHITELIST URL HERE")
Dim Response As HttpWebResponse = Request.GetResponse()
Using Reader As StreamReader = New StreamReader(Response.GetResponseStream())
S = Reader.ReadToEnd
End Using
Return S
End Function
This would make the attempt to contact the host and, if that attempt failed, throw an exception instead of return a string.
2) Throw a custom exception:
Public Function GetWhitelist(ByVal PageURL As String) As String
Dim S As String = ""
Try
Dim Request As HttpWebRequest = WebRequest.Create("WHITELIST URL HERE")
Dim Response As HttpWebResponse = Request.GetResponse()
Using Reader As StreamReader = New StreamReader(Response.GetResponseStream())
S = Reader.ReadToEnd
End Using
Catch ex As Exception
Debug.WriteLine("Start Program Error. Handle:0")
Throw New SomeCustomException(String.Format("Unable to contact host: {0}", PageURL), ex)
End Try
Return S
End Function
This would provide a more targeted exception instead of whatever comes out of the response reader, would provide useful runtime information about the error for logging and analysis (namely the runtime value of the PageURL), and takes a step toward hiding the implementation details from code outside of this object (since that code doesn't really care about the HttpWebRequest and HttpWebResponse, it just wants to know if the URL is good or not).
Remember that throwing an exception is a perfectly acceptable exit path for a function. It doesn't always have to return a value. An exception is an appropriate way of indicating an error condition. Your current implementation, however, "swallows" the exception and provides no indication to the consuming code that anything went wrong. Instead it returns a "magic value" of String.Empty which consuming code may or may not ignore.
Change your exception handler. The correct way to do this is to just try the request and handle the exception if it fails.
Trying to download file in code.
Current code:
Dim uri As New UriBuilder
uri.UserName = "xxx"
uri.Password = "xxx"
uri.Host = "xxx"
uri.Path = "xxx.aspx?q=65"
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(uri.Uri), HttpWebRequest)
request.AllowAutoRedirect = True
request = DirectCast(WebRequest.Create(DownloadUrlIn), HttpWebRequest)
request.Timeout = 10000
'request.AllowWriteStreamBuffering = True
Dim response As HttpWebResponse = Nothing
response = DirectCast(request.GetResponse(), HttpWebResponse)
Dim s As Stream = response.GetResponseStream()
'Write to disk
Dim fs As New FileStream("c:\xxx.pdf", FileMode.Create)
Dim read As Byte() = New Byte(255) {}
Dim count As Integer = s.Read(read, 0, read.Length)
While count > 0
fs.Write(read, 0, count)
count = s.Read(read, 0, read.Length)
End While
'Close everything
fs.Close()
s.Close()
response.Close()
Running this code and checking the response.ResponseUri indicates im being redirected back to the login page and not to the pdf file.
For some reason its not authorising access what could I be missing as Im sending the user name and password in the uri? Thanks for your help
You don't need all of that code to download a file from the net
just use the WebClient class and its DownloadFile method
you should check and see if the site requires cookies (most do), i'd use a packet analyzer and run your code and see exactly what the server is returning. use fiddler or http analyzer to log packets
With UWP, this has become a more pertinent question as UWP does not have a WebClient. The correct answer to this question is if you are being re-directed to the login page, then there must be an issue with your credentials OR the setting (or lack of) header for the HttpWebRequest.
According to Microsoft, the request for downloading is sent with the call to GetResponse() on the HttpWebRequest, therefore the downloaded file SHOULD be in the stream in the response (returned by the GetResponse() call mentioned above).
i am working in a project in that i want to read some data from the remote url can any one help me how to do this function
You can use a WebRequest to retrieve the XML from the remote site; then you can parse the contents into an XmlDocument object.
' Create a WebRequest to the remote site
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create("http://www.domain.com/fetch.xml")
' NB! Use the following line ONLY if the website is protected
request.Credentials = New System.Net.NetworkCredential("username", "password")
' Call the remote site, and parse the data in a response object
Dim response As System.Net.HttpWebResponse = request.GetResponse()
' Check if the response is OK (status code 200)
If response.StatusCode = System.Net.HttpStatusCode.OK Then
' Parse the contents from the response to a stream object
Dim stream As System.IO.Stream = response.GetResponseStream()
' Create a reader for the stream object
Dim reader As New System.IO.StreamReader(stream)
' Read from the stream object using the reader, put the contents in a string
Dim contents As String = reader.ReadToEnd()
' Create a new, empty XML document
Dim document As New System.Xml.XmlDocument()
' Load the contents into the XML document
document.LoadXml(contents)
' Now you have a XmlDocument object that contains the XML from the remote site, you can
' use the objects and methods in the System.Xml namespace to read the document
Else
' If the call to the remote site fails, you'll have to handle this. There can be many reasons, ie. the
' remote site does not respond (code 404) or your username and password were incorrect (code 401)
'
' See the codes in the System.Net.HttpStatusCode enumerator
Throw New Exception("Could not retrieve document from the URL, response code: " & response.StatusCode)
End If
Along with what #Jon Skeet said there's also the built-in WebClient:
Dim MyData As String
Try
Using WC As New System.Net.WebClient()
MyData = WC.DownloadString("http://www.example.com/text.xml")
End Using
Catch ex As Exception
'Error downloading
End Try
Have you tried using XDocument.Load or XmlDocument.Load?
If those don't do what you want, please give more details.
try this
http://www.codeproject.com/Tips/992109/Parsing-Reading-XML-from-URL-in-VB-NET
There are some useful examples provided with it. Hope this helps...