Empty response HTTPWebRequest: Windows Phone 8 - vb.net

I am making a Windows Phone app and I am trying to get JSON data from a URL. The user needs to be logged into the website (which hosts the JSON data) in order to get JSON data and I cannot use the Web Browser control to display the data and then extract the string since the browser doesn't recognize it (for some weird reason) and asks to search for an app on Store which can handle that JSON file type. (If I open the URL in desktop browser on my Windows PC, I can see the raw JSON data). I can't use normal HTTPWebRequest or WebClient as to get JSON data the login cookies needs to be set (the JSON data is user-specific) and I can't extract the cookies from Web browser control and use it with WebClient or HTTPWebRequest. So the best thing I can do is use a special internal instance of IWebRequestCreate that is used internally by the WebBrowser. By opening background HTTP requests with that class, the cookies get automatically set as if they were created/sent by the WebBrowser control. But my code is not returning the JSON data, I get blank response, as in the string resp is empty.
Below is the code:
Dim browser = New WebBrowser()
Dim brwhttp = GetType(WebRequestCreator).GetProperty("BrowserHttp")
Dim requestFactory = TryCast(brwhttp.GetValue(Browser, Nothing), IWebRequestCreate)
Dim uri = New Uri("http://api.quora.com/api/logged_in_user?fields=inbox,notifs,following,followers")
Dim req = requestFactory.Create(uri)
req.Method = "GET"
req.BeginGetResponse(New AsyncCallback(AddressOf request_Callback), req)
Private Sub request_Callback(asyncResult As IAsyncResult)
Dim webRequest As HttpWebRequest = DirectCast(asyncResult.AsyncState, HttpWebRequest)
Dim webResponse As HttpWebResponse = DirectCast(webRequest.EndGetResponse(asyncResult), HttpWebResponse)
Dim tempStream As New MemoryStream()
webResponse.GetResponseStream().CopyTo(tempStream)
Dim sr As New StreamReader(tempStream)
Dim resp As String = sr.ReadToEnd
End Sub
What's wrong?

I found that CopyTo can leave the Stream's pointer at the end of the buffer, you probably need to reset tempStream's pointer to the beginning before attempting to read it with the StreamReader, here's the code...
webResponse.GetResponseStream().CopyTo(tempStream);
tempStream.Seek(0, SeekOrigin.Begin);
Dim sr As New StreamReader(tempStream);

Related

WebClient.UploadData "The underlying connection was closed"

I'm trying to upload a file from an FTP site to Basecamp using the Basecamp API. I'm using a simple console application. Here's my code:
Try
Dim accountID As String = ConfigurationManager.AppSettings("BaseCampID")
Dim projectID As Integer = 9999999
Dim folderName As String = "XXXXX/XXXXX"
Dim fileName As String = "XXX.zip"
'The URL to access the attachment method of the API
Dim apiURL = String.Format("https://basecamp.com/{0}/api/v1/projects/{1}/attachments.json", accountID, projectID)
'Get the file from the FTP server as a byte array
Dim fileBytes As Byte() = GetFileBytes(String.Format("{0}\\{1}", folderName, fileName))
'Initialize the WebClient object
Dim client As New WebClient()
client.Headers.Add("Content-Type", "application/zip")
'Need to provide a user-agent with a URL or email address
client.Headers.Add("User-Agent", "Basecamp Upload (email#email.com)")
'Keep the connection alive so it doesn't close
client.Headers.Add("Keep-Alive", "true")
'Provide the Basecamp credentials
client.Credentials = New NetworkCredential("username", "password")
'Upload the file as a byte array to the API, and get the response
Dim responseStr As Byte() = client.UploadData(apiURL, "POST", fileBytes)
'Convert the JSON response to a BaseCampAttachment object
Dim attachment As BaseCampAttachment
attachment = JSonHelper.FromJSon(Of BaseCampAttachment)(Encoding.Default.GetString(responseStr))
Catch ex As Exception
Console.WriteLine(ex.Message)
Finally
Console.ReadLine()
End Try
But whenever it calls client.UploadData, I get the error message "The underlying connection was closed: The connection was closed unexpectedly." I ran into this issue earlier and thought I solved it by adding the "Keep-Alive" header, but it's not working anymore. The API works if I upload a local file with client.UploadFile, but I'd like to just upload the file from they byte array from the FTP rather than downloading the file locally then uploading it to Basecamp.
Any thoughts would be greatly appreciated. Thanks!
I never figured out what was wrong with the WebClient call, but I ended up using a Basecamp API wrapper from https://basecampwrapper.codeplex.com. That wrapper uses HTTPRequest and HTTPResponse instead of WebClient.UploadData. It's also much easier to just use that wrapper than to try writing my own code from scratch.

How to catch the post data sent to the redirect URL (VB 2010)

I am trying to login to one web page via my Windows Form application (VB 2010) and get some response, but I don't know how to do it.
The server service is described here:
https://api.developer.betfair.com/services/webapps/docs/display/1smk3cen4v3lu3yomq5qye0ni/Interactive+Login+from+a+Desktop+Application
On beginning I tried to insert WebBrowser Control to my form, but I read many articles, also here, that using BeforeNavigate2 event is an old way.
I would like to do this:
Send HTTP request to webserver including username and password
Catch the post data sent to the redirect URL
Read the POST request body an get loginStatus and productToken (SSOID)
This is my code:
Private Sub getPOST()
' Create a request for the URL
Dim request As WebRequest = _
WebRequest.Create("https://identitysso.betfair.com/view/login?product=82&url=https://www.betfair.com&username=abc&password=abc")
request.Method = "POST"
' Get the response
Dim response As HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
' Get the stream containing content returned by the server.
Dim dataStream As Stream = response.GetResponseStream()
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
Dim responseFromServer As String = reader.ReadToEnd()
' Display the content.
RichTextBox1.Text = responseFromServer
' Cleanup the streams and the response.
reader.Close()
dataStream.Close()
response.Close()
End Sub
But this code returns HTML code of page where are not loginStatus and SSOID.
I examined this page and discovered there are a few problems with your code:
First you are posting to the wrong URL because the www.betfair.com/view/login form has action="/api/login", not "/view/login". Second, you are passing your username/password in the QueryString rather than in POSTDATA... this form uses method=post. Third, you are not passing all required fields. This code snippet should remedy these described problems, as well as answer your question about obtaining the redirect URL:
Sub TestWebClient()
' Create a request for the URL
' You were using the wrong URL & passing the values improperly
' I also changed this to HttpWebRequest, used to be WebRequest.
Dim request As HttpWebRequest = WebRequest.Create("https://identitysso.betfair.com/api/login") '
request.Method = "POST"
' New code added
request.ContentType = "application/x-www-form-urlencoded"
' New code added, this prevents the following of the 302 redirect in the Location header
request.AllowAutoRedirect = False
' New code added, this sends the values as POSTDATA, which is contained inside the HTTP POST request rather than in the URL
Using writer As New StreamWriter(request.GetRequestStream)
writer.Write("username=abc")
writer.Write("&password=abc")
writer.Write("&login=true")
writer.Write("&redirectMethod=POST")
writer.Write("&product=82")
writer.Write("&url=https://www.betfair.com/")
writer.Write("&ioBlackBox=")
End Using
' Get the response
' This will print the Location header (aka "Redirect URL") on the screen for you
' Make decisions as needed.
Dim response As HttpWebResponse = request.GetResponse()
Console.WriteLine("HTTP RESPONSE HEADERS:")
For Each item In response.Headers
Console.WriteLine(item & "=" & response.Headers(item))
Next
Console.ReadLine()
End Sub

Anonymous HTTP Web Request

I created http request application to test my web site qulatiy (see below).
Dim Request As HttpWebRequest = WebRequest.Create(webAddress)
Dim Response As HttpWebResponse = Request.GetResponse()
Request.Method = "Get"
Dim Reader As New StreamReader(Response.GetResponseStream)
Dim Html As String = Reader.ReadToEnd()
In this case, I would like to create anonymous request without catching the response. How can I do that?
To do so, u have to get a little low level , working with sockets
TcpCient in this case
Sample code
Imports System.Net.Sockets
Module Module1
Sub Main()
Dim tcpcli = New TcpClient()
tcpcli.Connect("google.co.in", 80)
Dim stream As NetworkStream = tcpcli.GetStream()
Dim reqdata As String = String.Format("GET / HTTP/1.1{0}Host: www.google.co.in{0}Connection: Close{0}{0}", vbCrLf)
Dim reqbytes() As Byte = Text.Encoding.ASCII.GetBytes(reqdata)
stream.Write(reqbytes, 0, reqbytes.Length)
stream.Close()
stream.Dispose()
tcpcli.Close()
End Sub
End Module
Network capture via wireshark (no response received)
You can make a web request anonymously by using ProxySharp. It basically makes the web request behind a random vpn each time. This makes it look like the request is coming from a different IP address on each request.

How to use HttpWebRequest to download file

Trying to download file in code.
Current code:
Dim uri As New UriBuilder
uri.UserName = "xxx"
uri.Password = "xxx"
uri.Host = "xxx"
uri.Path = "xxx.aspx?q=65"
Dim request As HttpWebRequest = DirectCast(WebRequest.Create(uri.Uri), HttpWebRequest)
request.AllowAutoRedirect = True
request = DirectCast(WebRequest.Create(DownloadUrlIn), HttpWebRequest)
request.Timeout = 10000
'request.AllowWriteStreamBuffering = True
Dim response As HttpWebResponse = Nothing
response = DirectCast(request.GetResponse(), HttpWebResponse)
Dim s As Stream = response.GetResponseStream()
'Write to disk
Dim fs As New FileStream("c:\xxx.pdf", FileMode.Create)
Dim read As Byte() = New Byte(255) {}
Dim count As Integer = s.Read(read, 0, read.Length)
While count > 0
fs.Write(read, 0, count)
count = s.Read(read, 0, read.Length)
End While
'Close everything
fs.Close()
s.Close()
response.Close()
Running this code and checking the response.ResponseUri indicates im being redirected back to the login page and not to the pdf file.
For some reason its not authorising access what could I be missing as Im sending the user name and password in the uri? Thanks for your help
You don't need all of that code to download a file from the net
just use the WebClient class and its DownloadFile method
you should check and see if the site requires cookies (most do), i'd use a packet analyzer and run your code and see exactly what the server is returning. use fiddler or http analyzer to log packets
With UWP, this has become a more pertinent question as UWP does not have a WebClient. The correct answer to this question is if you are being re-directed to the login page, then there must be an issue with your credentials OR the setting (or lack of) header for the HttpWebRequest.
According to Microsoft, the request for downloading is sent with the call to GetResponse() on the HttpWebRequest, therefore the downloaded file SHOULD be in the stream in the response (returned by the GetResponse() call mentioned above).

how to read xml from remote url in vb.net

i am working in a project in that i want to read some data from the remote url can any one help me how to do this function
You can use a WebRequest to retrieve the XML from the remote site; then you can parse the contents into an XmlDocument object.
' Create a WebRequest to the remote site
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create("http://www.domain.com/fetch.xml")
' NB! Use the following line ONLY if the website is protected
request.Credentials = New System.Net.NetworkCredential("username", "password")
' Call the remote site, and parse the data in a response object
Dim response As System.Net.HttpWebResponse = request.GetResponse()
' Check if the response is OK (status code 200)
If response.StatusCode = System.Net.HttpStatusCode.OK Then
' Parse the contents from the response to a stream object
Dim stream As System.IO.Stream = response.GetResponseStream()
' Create a reader for the stream object
Dim reader As New System.IO.StreamReader(stream)
' Read from the stream object using the reader, put the contents in a string
Dim contents As String = reader.ReadToEnd()
' Create a new, empty XML document
Dim document As New System.Xml.XmlDocument()
' Load the contents into the XML document
document.LoadXml(contents)
' Now you have a XmlDocument object that contains the XML from the remote site, you can
' use the objects and methods in the System.Xml namespace to read the document
Else
' If the call to the remote site fails, you'll have to handle this. There can be many reasons, ie. the
' remote site does not respond (code 404) or your username and password were incorrect (code 401)
'
' See the codes in the System.Net.HttpStatusCode enumerator
Throw New Exception("Could not retrieve document from the URL, response code: " & response.StatusCode)
End If
Along with what #Jon Skeet said there's also the built-in WebClient:
Dim MyData As String
Try
Using WC As New System.Net.WebClient()
MyData = WC.DownloadString("http://www.example.com/text.xml")
End Using
Catch ex As Exception
'Error downloading
End Try
Have you tried using XDocument.Load or XmlDocument.Load?
If those don't do what you want, please give more details.
try this
http://www.codeproject.com/Tips/992109/Parsing-Reading-XML-from-URL-in-VB-NET
There are some useful examples provided with it. Hope this helps...