i am using a stream reader to get the HTML of some page, but there are lines that i want to ignore, such as if a line starts with <span>
any advice?
Here is my function
Public Function GetPageHTMLReaderNoPrx(ByVal address As Uri) As StreamReader
Dim request As HttpWebRequest
Dim response As HttpWebResponse = Nothing
Dim reader As StreamReader
Try
request = DirectCast(WebRequest.Create(address), HttpWebRequest)
response = DirectCast(request.GetResponse(), HttpWebResponse)
Select Case CType(response, Net.HttpWebResponse).StatusCode
Case 200
reader = New StreamReader(response.GetResponseStream(), Encoding.Default)
Case Else
MsgBox(CType(response, Net.HttpWebResponse).StatusCode)
End Select
Catch
If Not response Is Nothing Then response.Close()
End Try
Return reader
End Function
this is how the HTML looks like
<tr>Text
<span>show all</span>
</tr>
If you insist on using strings, you could do something like this:
Do
Dim line As String = reader.ReadLine()
If line Is Nothing Then Exit Do 'end of stream
If line.StarsWith("<span>") Then Exit Do 'ignore this line
'otherwise do some processing here
'...
Loop
But this approach is not stable - any minor change in the input HTML can break your flow.
More elegant solution would be using XElement:
Dim xml = <tr>Text
<span>show all</span>
</tr>
xml.<span>.Remove()
MsgBox(xml.Value.Trim)
Related
I'm making a web request that completes successfully most of the time (target$ is a URL). But occasionally my code throws a valid exception, 404 not found, if the URL target$ doesn't exist, and execution stops. The code:
Sub scrape(target$)
Dim request As WebRequest = WebRequest.Create(target$)
Dim response As WebResponse = request.GetResponse()
Dim dataStream As Stream = response.GetResponseStream()
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
Dim responseFromServer As String = reader.ReadToEnd()
txtResponse.Text = ""
txtResponse.Text = responseFromServer
' Clean up the streams and the response.
reader.Close()
response.Close()
end sub
The exception, if thrown, happens in the second line, "Dim response...". So I tried adding a "try-catch" as shown.
Sub scrape(target$)
Dim request As WebRequest = WebRequest.Create(target$)
Dim response As WebResponse = request.GetResponse()
Try
Dim dataStream As Stream = response.GetResponseStream()
Catch
exflag = True
End Try
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
Dim responseFromServer As String = reader.ReadToEnd()
txtResponse.Text = ""
txtResponse.Text = responseFromServer
' Clean up the streams and the response.
reader.Close()
response.Close()
end sub
But now when I try to compile the code, VisualStudio tells me that "datastream is not declared" and the compile fails.
What am I doing wrong and how do I catch the exception when it's thrown?
Thanks...
I am unable to get the HttpWebRequest to work properly with OpenWeatherMap.
When I try out the URL from the browser I get the data. However, when I am sending it from the program I'm getting a message with code id. Like this:
"message":"","cod":"404"
What Am I doing wrong?
VB.NET Code:
Private Shared AppID As String = "add_app_id_Here"
Public Shared Function GetWeather(ByVal location As String) As List(Of WeatherDetails)
Dim url = String.Format _
("http://api.openweathermap.org/data/2.5/forecast/daily?q={0}&type=accurate&mode=xml&units=metric&cnt=3&appid={1}",
location, AppID)
Try
Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
request.AuthenticationLevel = Net.Security.AuthenticationLevel.None
Dim response As WebResponse = request.GetResponse()
'The body of the request is sent here
Dim responseReader As New StreamReader(response.GetResponseStream())
Dim responseInfo As String = responseReader.ReadToEnd()
responseReader.Close()
response.Close()
If Not (responseInfo.Contains("message") And responseInfo.Contains("cod")) Then
Dim xEl = XElement.Load(New System.IO.StringReader(responseInfo))
Return GetWeatherInfo(xEl)
Else
Return New List(Of WeatherDetails)
End If
Catch ex As Exception
Return New List(Of WeatherDetails)
End Try
End Function
After breaking my head I found that his method HttpWebRequest is not tolerant to special characters or non printable characters.
Thus in the URL I had to hardcode the "%27" character and it solved the problem.
So I feel stupid for asking probably a easy question but I'm not very familiar with .NET and I've been googling for a while now.
I'm looking to take in data from a web API and be able to iterate over it. The data looks like this
[
{"id":"5", "date":"01-01-2014"},
{"id":"90", "date":"05-01-2013"}
]
What I've got so far:
Private Sub newGetData()
Dim request As HttpWebRequest
Dim response As HttpWebResponse = Nothing
Dim reader As StreamReader
Try
request = DirectCast(WebRequest.Create("http://somesite.com"), HttpWebRequest)
response = DirectCast(request.GetResponse(), HttpWebResponse)
reader = New StreamReader(response.GetResponseStream())
Dim rawresp As String
rawresp = reader.ReadToEnd()
Dim jResults As JObject = JObject.Parse(rawresp)
Dim results As List(Of JToken) = jResults.Children().ToList
For Each item As JProperty In results
item.CreateReader()
MsgBox(item.name)
Next
'usernameTextbox.text = jResults("name").ToString()
'placenameTextbox.text = jResults("place")("name").ToString()
Catch ex As Exception
MsgBox(ex.ToString)
Finally
If Not response Is Nothing Then response.Close()
End Try
End Sub
I've tried a bunch of things but it always fails on the loop. How do I go about doing this?
EDIT: The error message for this one
Newtonsoft.Json.JsonReaderException: Error reading JObject from JsonReader. Current JsonReader item is not an object: StartArray. Path '', line 1, position 1.
at Newtonsoft.Json.Linq.JObject.Load(JsonReader reader)
at Newtonsoft.Json.Linq.JObject.Parse(String json)
at Forecasting.Form1.newGetData() in
I am fetching 3 line of text from a website with my vb.net application and i only want to show the second line in my my label when i press a button. The code i currently have show me all the 3 lines. How can i only display the 2nd line.
Dim webAddress As String = "Website"
Dim reader As StreamReader
Dim request As WebRequest
Dim response As WebResponse
Dim data As String = ""
Try
request = WebRequest.Create(webAddress)
request.Timeout = 30000
response = request.GetResponse()
reader = New StreamReader(response.GetResponseStream())
data = reader.ReadToEnd
Catch ex As Exception
MsgBox(ex.Message)
End Try
You may do this after ReadToEnd:
Label1.Text = Split(data, VBCrLf)(1)
Read individual lines:
reader.ReadLine() 'read and discard the first line
data = reader.ReadLine() 'read the line you want
reader.ReadToEnd 'read and discard the rest
I am attempting to perform a FTP Put function to an AS/400 IBM Mainframe with VB.NET. I am able to upload a file however, I need to be able to capture each output response from the mainframe for logging purposes. In short capture what prints out on the cmd screen if I were to perform the FTP manually. Any suggestions would be greatly appreciated.
Depending on the library you are using, you should be able to get some kind of response object or string from the FTP server for each command you submit. You can then parse these responses and dump them into a file/destination/source of your choosing.
EDIT: Since you're using the FTPWebRequest/Response library, you'll want to have your FTPWebRequest object dump its results into the FTPWebResponse object and then read the entire stream with code something like this:
Dim request As FtpWebRequest = DirectCast(WebRequest.Create(serverUri), FtpWebRequest)
request.Method = WebRequestMethods.Ftp.ListDirectory
Dim response As FtpWebResponse = DirectCast(request.GetResponse(), FtpWebResponse)
Dim responseStream As Stream = Nothing
Dim readStream As StreamReader = Nothing
Try
responseStream = response.GetResponseStream()
readStream = New StreamReader(responseStream, System.Text.Encoding.UTF8)
If readStream IsNot Nothing Then
Console.WriteLine(readStream.ReadToEnd())
End If
Console.WriteLine("List status: " & response.StatusDescription)
Finally
If readStream IsNot Nothing Then
readStream.Close()
End If
If response IsNot Nothing Then
response.Close()
End If
End Try
Return True
End Function
You should be able to tailor this code to your own in order to retrieve the response details you need.