I am writing code where I am trying to grab the HTML from a DNS report online (http://viewdns.info/dnsreport/?domain=google.com), but I am having some issues. The one line of the HTML file (Line 231) that I actually need is cutting itself off after around 680 characters. All of the lines after the important one are reading correctly, however. The code for grabbing the HTML is shown below, and I have tried it in two separate ways.
This is the first way I tried:
Public Function getWebResourceData(ByVal strURL As String) As String
Dim webClient As New System.Net.WebClient
Dim result As String = webClient.DownloadString("http://viewdns.info/dnsreport/?" & TextBox1.Text)
return result
End Function
And this is the second:
Public Function getWebResourceData(ByVal strURL As String) As String
Dim rt As String = ""
Dim wRequest As WebRequest
Dim wResponse As WebResponse
Dim SR As StreamReader
wRequest = WebRequest.Create(strURL)
wResponse = wRequest.GetResponse
SR = New StreamReader(wResponse.GetResponseStream)
rt = SR.ReadToEnd
SR.Close()
return rt
End Function
Im really not sure what else could be wrong at this point. I have also tried saving the result to a text file to see if that was the issue, but that was incorrect as well. I have looked into the hex codes for the area where the string is stopping, but there isn't anything out of the ordinary. The split occurs between the back to back alligator brackets (shown as parentheses) here: (/tr)(tr)
But there are numerous sets of these tags throughout the HTML that there are no issues with.
Both of your functions don't return what they have read. I have tested the second one and it works correctly.
Sub Main
Dim ret = getWebResourceData("http://viewdns.info/dnsreport/?domain=google.com")
Console.WriteLine(ret.Length)
' Output = 21605
End Sub
Public Function getWebResourceData(ByVal strURL As String) As String
Dim rt As String = ""
Dim wRequest As WebRequest
Dim wResponse As WebResponse
Dim SR As StreamReader
wRequest = WebRequest.Create(strURL)
wResponse = wRequest.GetResponse
SR = New StreamReader(wResponse.GetResponseStream)
rt = SR.ReadToEnd
SR.Close()
return rt
End Function
Related
I have tried de-serializing with suggested code in this forum.
I use a Google Translate API. It returns a JSON string.
I use Newtonsoft.Json to de-serialize.
My code does not work for foreign language translations, where the string to deserialise is more than one byte.
The code is shown below:
Public Function getGoogleTranslate(myIncomingText As String) As String
Dim myUrlString As String
Dim myLanguageFrom As String
Dim myLanguageTo as string
Dim myTextFrom As String
Dim myNewString As String
myLanguageFrom = "en"
myLanguageTo = "fr"
myTextFrom = myIncomingText
myUrlString = "https://translate.googleapis.com/translate_a/single?client=gtx&sl=auto&tl="
myUrlString &= myLanguageTo
myUrlString &= "&hl="
myUrlString &= myLanguageFrom
myUrlString &= "&dt=t&dt=bd&dj=1&source=icon&q="
myUrlString &= myIncomingText
Dim myWebClient As New System.Net.WebClient
Dim myDowloadString As String = myWebClient.DownloadString(myUrlString)
Dim myJsonFile As Newtonsoft.Json.Linq.JObject = Newtonsoft.Json.Linq.JObject.Parse(myDowloadString)
myNewString = myJsonFile.SelectToken("sentences[0]").SelectToken("trans")
Return myNewString
End Function
Everything works. I call the API using the URL specified as "MyURLString".
The returned line is returned into the string myDownloadString.
Investigation of this string is that the translated text is returned correctly.
After the resulting string is parsed, only characters in the ASCII range are decoded as expected, not characters used in other languages.
It could be the myNewString variable, which is declared as a standard string.
However, if I cut and paste the translated text into a TextBox on the web page, the special characters are accepted and stored in a SQL table correctly.
It gets even more complex when translation to "ru" (Russian) or "zh" (simplified Chinese).
I have never worked with a different language character set. So I am flying blind, with only this forum for help.
The data you're downloading is UTF8 encoded.
You can decode it using Encoding.UTF8.GetString(), downloading the results as a byte array, using the DownloadData() method instead of DonloadString().
' [...]
Dim data As Byte()
Dim jsonResult as String = String.Empty
Using client As New WebClient()
data = client.DownloadData(myUrlString)
jsonResult = Encoding.UTF8.GetString(data)
End Using
' Deserialize the jsonResult object
Unfortunately, WebClient doesn't care at all about the encoding of the incoming string, it always uses the Local encoding, unless otherwise specified, setting its Encoding Property. Or using other means, as shown here. You need to know what the encoding is beforehand, though. Or you could read the Encoding from the underlying WebResponse object.
Option 2: use HttpClient instead of WebClient. This class handles the encoding specified by the remote source:
' Declare a static (`Shared`) HttpClient object as a Field
Private Shared client As New HttpClient()
' Make an async method:
Private async Function GetGoogleTranslation(textToTranslate As String) As String
' Declare your local variables
' [...]
Dim jsonResult as String = String.Empty
Using response = Await client.GetAsync(myUrlString)
If response.IsSuccessStatusCode Then
jsonResult = Await response.Content.ReadAsStringAsync()
End If
End Using
' Deserialize and get the results you need
Dim result = [...] ' Deserialize the result
Return result
End Function
I have managed to crack this. It was not a JSON problem after all. The problem was in the programmer. I added one line of code to my solution:
myWebClient.Encoding = System.Text.UTF8Encoding.UTF8
NewtonSoft does handle the parse correctly, even if the incoming string is in UTF8.
Thanks for your answer Jimi.
The working code to call the Google translate API from VB.NET. Note: I have tested this with French and German translations but not with Chinese yet.
Public Function getGoogleTranslate(myIncomingText As String) As String
Dim myUrlString As String
Dim myLanguageFrom As String
Dim myLanguageTo as string
Dim myTextFrom As String
Dim myNewString As String
myLanguageFrom = "en"
myLanguageTo = "fr"
myTextFrom = myIncomingText
myUrlString = "https://translate.googleapis.com/translate_a/single?client=gtx&sl=auto&tl="
myUrlString &= myLanguageTo
myUrlString &= "&hl="
myUrlString &= myLanguageFrom
myUrlString &= "&dt=t&dt=bd&dj=1&source=icon&q="
myUrlString &= myIncomingText
Dim myWebClient As New System.Net.WebClient
myWebClient.Encoding = System.Text.UTF8Encoding.UTF8
Dim myDowloadString As String = myWebClient.DownloadString(myUrlString)
Dim myJsonFile As Newtonsoft.Json.Linq.JObject = Newtonsoft.Json.Linq.JObject.Parse(myDowloadString)
myNewString = myJsonFile.SelectToken("sentences[0]").SelectToken("trans")
Return myNewString
End Function
If I can find line that contains word in a file
File.ReadAllLines(html).FirstOrDefault(Function(x) x.Contains("something"))
How can I find all lines that contains in a string
for example I made an webresponse
Dim rt As String = "http://www.somesaite.com"
Dim wRequest As WebRequest
Dim WResponse As WebResponse
Dim SR As StreamReader
wRequest = FtpWebRequest.Create(rt)
WResponse = wRequest.GetResponse
SR = New StreamReader(WResponse.GetResponseStream)
rt = SR.ReadToEnd
How to find lines that contains in rt ?
You could read all the text that the StreamReader gives you, and then you could split that by the Environment.NewLine character(s). Then you should just be able to use the lambda expression you first mentioned (as the File.ReadAllLines() method returns an array of strings).
Dim FoundLine As String = SR.ReadToEnd().Split(Environment.NewLine).FirstOrDefault(Function(x) x.Contains("something"))
I am unable to get the HttpWebRequest to work properly with OpenWeatherMap.
When I try out the URL from the browser I get the data. However, when I am sending it from the program I'm getting a message with code id. Like this:
"message":"","cod":"404"
What Am I doing wrong?
VB.NET Code:
Private Shared AppID As String = "add_app_id_Here"
Public Shared Function GetWeather(ByVal location As String) As List(Of WeatherDetails)
Dim url = String.Format _
("http://api.openweathermap.org/data/2.5/forecast/daily?q={0}&type=accurate&mode=xml&units=metric&cnt=3&appid={1}",
location, AppID)
Try
Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
request.AuthenticationLevel = Net.Security.AuthenticationLevel.None
Dim response As WebResponse = request.GetResponse()
'The body of the request is sent here
Dim responseReader As New StreamReader(response.GetResponseStream())
Dim responseInfo As String = responseReader.ReadToEnd()
responseReader.Close()
response.Close()
If Not (responseInfo.Contains("message") And responseInfo.Contains("cod")) Then
Dim xEl = XElement.Load(New System.IO.StringReader(responseInfo))
Return GetWeatherInfo(xEl)
Else
Return New List(Of WeatherDetails)
End If
Catch ex As Exception
Return New List(Of WeatherDetails)
End Try
End Function
After breaking my head I found that his method HttpWebRequest is not tolerant to special characters or non printable characters.
Thus in the URL I had to hardcode the "%27" character and it solved the problem.
I'm making a Windows Phone 8 app. I have the latitude and longitude of the target location. I need to get the Two Letter ISO country code of target location.
I'm using the following code to make it happen.
'Dim address As String = puri
'Dim client As WebClient = New WebClient()
'Dim reader As StreamReader = New StreamReader(address)
'code = reader.ReadToEnd
Dim inStream As StreamReader
Dim wr As WebRequest
Dim webresponse As WebResponse
wr = WebRequest.Create(puri)
webresponse = wr.GetResponse()
inStream = New StreamReader(webresponse.GetResponseStream())
code = inStream.ReadToEnd()
where puri(in the commented code) is the address of the webservice in string format.
When trying the commented code, the error I'm getting is that string cannot be converted to system.uri format. (address)
When trying the uncommented code, I get an error which says, getresponse is not a member of class system.net.webrequest()
I guess with the updates to .NET the code changed, but I couldn't find anything current on the topic.
URI = http://api.geonames.org/countryCode?lat=17.60890&lng=76.98966&username=demo
I think you should use WebClient Class instead of a WebRequest. It is simpler and faster. Here is a simple example:
Dim WebCL As New WebClient
Dim DownLoadedText As String = String.Empty
Try
DownLoadedText = WebCL.DownloadString("Your Url")
' Do something
Catch ex As Exception
Throw New Exception("Oops!! ERROR has occured, something is wrong with your address")
End Try
I'm requesting remote SOAP web-service but all operation (from click search button to render interface with answer) took almost two minutes, it's too long. So I wonder if there any possible way to improve performance of the current code.
Operation that parse xml and read data to database working quite well, problem only about reading answer from stream.
Public Shared Function CallWebService(ByVal an As String, ByVal xmlcommand As String) As String
Dim _url = "http://testapi.interface-xml.com/appservices/ws/FrontendService"
Dim soapEnvelopeXml As XmlDocument = CreateSoapEnvelope(xmlcommand)
Dim webRequest As HttpWebRequest = CreateWebRequest(_url, an)
webRequest.Proxy = System.Net.WebRequest.DefaultWebProxy
InsertSoapEnvelopeIntoWebRequest(soapEnvelopeXml, webRequest)
Dim asyncResult As IAsyncResult = webRequest.BeginGetResponse(Nothing, Nothing)
asyncResult.AsyncWaitHandle.WaitOne()
Dim soapResult As String
Using webResponse As WebResponse = webRequest.EndGetResponse(asyncResult)
Using bs As New BufferedStream(webResponse.GetResponseStream())
Using rd As New StreamReader(bs)
soapResult = rd.ReadLine()
Return soapResult
End Using
End Using
End Using
End Function
Here is solution!
Public Shared Function CallWebService(ByVal an As String, ByVal xmlcommand As String) As String
Dim _url = "http://testapi.interface-xml.com/appservices/ws/FrontendService"
Dim soapEnvelopeXml As XmlDocument = CreateSoapEnvelope(xmlcommand)
Dim webRequest As HttpWebRequest = CreateWebRequest(_url, an)
webRequest.Proxy = System.Net.WebRequest.DefaultWebProxy
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate")
InsertSoapEnvelopeIntoWebRequest(soapEnvelopeXml, webRequest)
Dim asyncResult As IAsyncResult = webRequest.BeginGetResponse(Nothing, Nothing)
asyncResult.AsyncWaitHandle.WaitOne()
Dim soapResult As String
Using webResponse As WebResponse = webRequest.EndGetResponse(asyncResult)
Using bs As New BufferedStream(webResponse.GetResponseStream())
Using gz As New GZipStream(bs, CompressionMode.Decompress)
Using rd As New StreamReader(gz)
soapResult = rd.ReadLine()
Return soapResult
End Using
End Using
End Using
End Using
End Function