i want to using webrequest codes then adding google search result URL's listbox1. But i can't codes gives error.
Try
Dim adres As String = "https://www.google.com/search?q=" + TextBox1.Text
Dim istek As WebRequest = HttpWebRequest.Create(adres)
Dim cevap As WebResponse
cevap = istek.GetResponse()
Dim donenBilgiler As StreamReader = New StreamReader(cevap.GetResponseStream())
Dim gelen As String = donenBilgiler.ReadToEnd()
Dim titleIndexBaslangici As Integer = gelen.IndexOf("<link href") + 2
Dim titleIndexBitisi As Integer = gelen.Substring(titleIndexBaslangici).IndexOf(">")
ListBox1.Items.Add(gelen.Substring(titleIndexBaslangici, titleIndexBitisi))
Catch ex As Exception
MsgBox(ex.Message)
End Try
First, are you familliar with API, because that's what you need ! You might work this out with the way you want to make it but its really bad and no one will recommend you to continue like this!
What you need to look for is API, (google search API)! API "only" purpose is to access to some data (in database) with easy Http route (well documented), try it out!
If you keep trying to do it in your own way, the best result that you will get is a really bad html page that you will need to parse and you don't want that !
Related
I'm writing an app in vb.net that needs public IP address in text only format. I'd knew there is lots of site that give you your IP in text format. But where is always a chance of being closed or getting out of service. But the Google will not ever stop! Now I want to get my ip from google searching. For example if you search "my ip" in google it will bring your ip like this:
Sample of search
Is anyway to get IP from Google?
Thanks guys but I found a way:
At first import some namespaces:
Imports System.Net
Imports System.Text.RegularExpressions
Now lets write a function:
Dim client As New WebClient
Dim To_Match As String = "<div class=""_h4c _rGd vk_h"">(.*)"
Dim recived As String = client.DownloadString("https://www.google.com/search?sclient=psy-ab&site=&source=hp&btnG=Search&q=my+ip")
Dim m As Match = Regex.Match(recived, To_Match)
Dim text_with_divs As String = m.Groups(1).Value
Dim finalize As String() = text_with_divs.Split("<")
Return finalize(0)
It's now working and live!
Hard-coded Div Class names make me a bit nervous because they can easily change at any time, so, I expanded on Hirod Behnam's example a little bit.
I removed the Div Class pattern, replacing it with a simpler IP address search, and it'll return only the first one found, which for this search, should be the first one shown on the page (your external IP).
That also removed the need for splitting the results into the array, and those related vars. I also simplified the Google search string to the bare minimum.
It'd still probably be a nice touch to include a timeout or two for the .DownloadString() and the .Match() respectively, if speed is of the essence.
Private Function GetExternalIP() As String
Dim m As Match = Match.Empty
Try
Dim wClient As New System.Net.WebClient
Dim strURL As String = wClient.DownloadString("https://www.google.com/search?q=my+ip")
Dim strPattern As String = "\b(?:(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b"
' Look for the IP
m = Regex.Match(strURL, strPattern)
Catch ex As Exception
Debug.WriteLine(String.Format("GetExternalIP Error: {0}", ex.Message))
End Try
' Failed getting the IP
If m.Success = False Then Return "IP: N/A"
' Got the IP
Return m.value
End Function
I need to get some text from this web page. I want to use the trade feed for my program to analyse the sentiment of the markets.
I used the browser control and the get element command but its not working. The problem is that whenever my browser starts to open the page I get Java scripts errors.
I tried with DOM but seems that i dont quite understand what i need to do :)
Here is the code:
Dim code As String
Using client As New WebClient
code = client.DownloadString("http://openbook.etoro.com/ahanit/#/profile/Trades/")
End Using
Dim htmlDocument As IHTMLDocument2 = New HTMLDocument(code)
htmlDocument.write(htmlDocument)
Dim allElements As IHTMLElementCollection = htmlDocument.body.all
Dim allid As IHTMLElementCollection = allElements.tags("id")
Dim element As IHTMLElement
For Each element In allid
element.title = element.innerText
MsgBox(element.innerText)
Next
Update: So I tried the HTML Agility pack, as suggested in the comments, and I am stuck again on this code
Dim plain As String = String.Empty
Dim htmldoc As New HtmlAgilityPack.HtmlDocument
htmldoc.LoadHtml("http://openbook.etoro.com/ahanit/#/profile/Trades/")
Dim goodnods As HtmlAgilityPack.HtmlNodeCollection = htmldoc.DocumentNode.SelectNodes("THE PROBLEM")
For Each node In goodnods
TextBox1.Text = htmldoc.DocumentNode.InnerText
Next
Any advice what to now?
Ok I think I know what the problem is somehow the div that I need is hidden and its not loaded when I load the web page just the source code. Does someone knows how to load all the hidden divs ??
Here is my new code
Dim doc As New HtmlAgilityPack.HtmlDocument
Dim web As New HtmlWeb
doc = web.Load("http://openbook.etoro.com/ahanit/#/profile/Trades/")
Dim nodes As HtmlNode = doc.GetElementbyId("feed-items")
Dim id As String = nodes.WriteTo()
TextBox1.Text = TextBox1.Text & vbCrLf & id
user1336635,
Welcome to SO! Something you might try is to check out his source code, figure out what javascript function is populating the field you want (using firebug - I assume it's the one that "trades result in profit" next to it), and then embedding that script into a web page that your webbrowser control loads. That's where I'd try to start. I checked his source code and searched for "trades result in profit" and didn't find anything which leads me to believe hunting for the element 'might' not be possible. Just a starting place until someone with more experience with this chimes in!! Best!
-sf
I am trying to loop through an array and perform an httpwebrequest in each iteration.
The code seems to work, however it pauses for a while (eg alot longer than the set timeout. Tried setting that to 100 to check and it still pauses) after every 10 or so iterations, then carries on working.
Here is what i have so far:
For i As Integer = 0 To numberOfProxies - 1
Try
'create request to a proxyJudge php page using proxy
Dim request As HttpWebRequest = HttpWebRequest.Create("http://www.pr0.net/deny/azenv.php")
request.Proxy = New Net.WebProxy(proxies(i)) 'select the current proxie from the proxies array
request.Timeout = 10000 'set timeout
Dim response As HttpWebResponse = request.GetResponse()
Dim sr As StreamReader = New StreamReader(response.GetResponseStream())
Dim pageSourceCode As String = sr.ReadToEnd()
'check the downloaded source for certain phrases, each identifies a type of proxy
'HTTP_X_FORWARDED_FOR identifies a transparent proxy
If pageSourceCode.Contains("HTTP_X_FORWARDED_FOR") Then
'delegate method for cross thread safe
UpdateListbox(ListBox3, proxies(i))
ElseIf pageSourceCode.Contains("HTTP_VIA") Then
UpdateListbox(ListBox2, proxies(i))
Else
UpdateListbox(ListBox1, proxies(i))
End If
Catch ex As Exception
'MessageBox.Show(ex.ToString) used in testing
UpdateListbox(ListBox4, proxies(i))
End Try
completedProxyCheck += 1
lblTotalProxiesChecked.CustomInvoke(Sub(l) l.Text = completedProxyCheck)
Next
I have searched all over this site and via google, and most responses to this type of question say the response must be closed. I have tried a using block, eg:
Using response As HttpWebResponse = request.GetResponse()
Using sr As StreamReader = New StreamReader(response.GetResponseStream())
Dim pageSourceCode As String = sr.ReadToEnd()
'check the downloaded source for certain phrases, each identifies a type of proxy
'HTTP_X_FORWARDED_FOR identifies a transparent proxy
If pageSourceCode.Contains("HTTP_X_FORWARDED_FOR") Then
'delegate method for cross thread safe
UpdateListbox(ListBox3, proxies(i))
ElseIf pageSourceCode.Contains("HTTP_VIA") Then
UpdateListbox(ListBox2, proxies(i))
Else
UpdateListbox(ListBox1, proxies(i))
End If
End Using
End Using
And it makes no difference (though i may have implemented it wrong) As you can tell im very new to VB or any OOP so its probably a simple problem but i cant work it out.
Any suggestions or just tips on how to diagnose these types of problems would be really appreciated.
EDIT:
Now im thoroughly confused. Does the try catch statement automatically close the response, or do i need to put something in Finally? If so, what? i cant use response.close() because its declared in the try block.
Perhaps im just using really badly structured code and there is a much better way to do this? Or something else is causing the pause/hang?
Yeah, you need to close the response after you are done with it, as .net enforces a maximum number of concurrent requests
so just add
response.close()
at the end of your code block
Because, it's a very difficult to write code in comment I will continue as answer.
For i As Integer = 0 To numberOfProxies - 1
Dim response As HttpWebResponse
Try
'create request to a proxyJudge php page using proxy
Dim request As HttpWebRequest = HttpWebRequest.Create("http://www.pr0.net/deny/azenv.php")
request.Proxy = New Net.WebProxy(proxies(i)) 'select the current proxie from the proxies array
request.Timeout = 10000 'set timeout
response = request.GetResponse()
Dim sr As StreamReader = New StreamReader(response.GetResponseStream())
Dim pageSourceCode As String = sr.ReadToEnd()
'check the downloaded source for certain phrases, each identifies a type of proxy
'HTTP_X_FORWARDED_FOR identifies a transparent proxy
If pageSourceCode.Contains("HTTP_X_FORWARDED_FOR") Then
'delegate method for cross thread safe
UpdateListbox(ListBox3, proxies(i))
ElseIf pageSourceCode.Contains("HTTP_VIA") Then
UpdateListbox(ListBox2, proxies(i))
Else
UpdateListbox(ListBox1, proxies(i))
End If
Catch ex As Exception
'MessageBox.Show(ex.ToString) used in testing
UpdateListbox(ListBox4, proxies(i))
Finally
response.Close()
End Try
completedProxyCheck += 1
lblTotalProxiesChecked.CustomInvoke(Sub(l) l.Text = completedProxyCheck)
Next
I'm using the HTMLAgilityPack to parse HTML pages. However at some point I try to parse wrong data (in this specific case an image), which ofc fails for obvious reasons.
Private Sub parseHtml(ByVal content As String, ByVal url As String)
Try
Dim contentHash As String = hashGenerator.ComputeHash(content, "SHA1")
Dim doc As HtmlDocument = New HtmlDocument()
doc.Load(New StringReader(content))
Dim root As HtmlNode = doc.DocumentNode
Dim anchorTags As New List(Of String)
For Each link As HtmlNode In root.SelectNodes("//a")
cururl = link.OuterHtml
If link.Attributes("href") Is Nothing Then Continue For
If Uri.IsWellFormedUriString(link.Attributes("href").Value, UriKind.Absolute) Then
urlQueue.Enqueue(link.Attributes("href").Value)
Else
Dim myUri As New Uri(url)
urlQueue.Enqueue(myUri.Scheme & "://" & myUri.Host & link.Attributes("href").Value)
End If
Next
Catch ex As Exception
MsgBox(ex.Message, MsgBoxStyle.Critical, "Error (parseHtml(" & url & "))")
End Try
End Sub
The error I get is:
A first chance exception of type
'System.NullReferenceException'
occurred in Webcrawler.exe Object
reference not set to an instance of an
object.
On the content I try to parse:
�����Iޥ�+�: 8�0�x�
How to check whether the content is 'parse-able' before trying to parse it to prevent the error?
For now it is an image which makes an error popup however I think it might be just anything which isn't (x)html.
Thanks in advance ow great community :)
You need to check the returned content-type header before trying to parse the returned data.
For an HTML page this should be text/html, for XHTML is would be application/xhtml+xml.
If you only have the content (If you can't have access to original HTTP headers like Oded suggested), you could assume a good HTML string should contain at least a "<" character within, say, the 10 first characters of the string.
Of course, there is no guarantee and you will still need to handle the extreme cases, but this should discard most garbage or unexpected content types, and will let specific encoding bytes pass fine (like UTF-8 byte order mark, etc...).
i have a application thats writen with the pylons framework. Now i want to call some controllers from a vb.net application. How should i do this?
I've tried it like this:
Dim webclient As New WebClient
Dim dataStream As IO.Stream = webclient.OpenRead("http://192.168.0.20:5000/controller/default")
Dim reader As New StreamReader(dataStream)
Dim responseFromServer As String = reader.ReadToEnd()
Dim erg As String = responseFromServer.ToString
reader.Close()
dataStream.Close()
But instead of an json object that is generate by the pylons controller, i'll get the html code for the page which is reachable under "http://192.168.0.20:5000"
Any help would be great!
Cheers,
Nico
You're probably requesting either the wrong content type or the wrong URL.
Make sure the URL is correct, or try this code:
Dim webclient As New WebClient
webclient.Headers.Add(HttpRequestHeader.ContentType, "test/json")
Dim erg As String = webclient.DownloadString("http://192.168.0.20:5000/controller/default")
As I demonstrated, you should use the DownloadString method instead of manually using a StreamReader.
Thank you for the answer, but this brings me the same result.
I figured out that you have to first login to the page. That means i have to call another controller, which is responsible for the login. For that i have to add params in the post.
How i do this in vb.net?
Cheers Nico