WordPress Post Scraper visual basic - vb.net

Question

I need following. I need to scrape posts from mine WordPress blog and show them in VB application. But news must be fresh, when user clicks "Refresh" it gets new content from website. So, posts from first page and link to it.
Any way to do this? Is that even possible?

Answer 1

This is very possible and I will give you three possible solutions.
The first solution is to use Telerik's (free!) Testing Framework http://www.telerik.com/teststudio/testing-framework. It is meant to be used for testing your website for flaws and whatnot but it makes for an excellent scraper. Once you get to know how to use the syntax this is most likely the fastest and easiest solution but it also has the most overhead. It uses browser addons to allow you to "control" Mozilla Firefox, Google Chrome, Apple Safari and Microsoft Internet Explorer. The main disadvantage with this one is that you must install the setup package on all computers that you wish to use this on. If you are only building a simple scraper to get data from your own blog and you will only run this on one machine this is probably the best way to go.
The next scraper I will mention is called Watin and you can get it from http://watin.org/. I personally prefer this one to Telerik's because it only requires a few dlls to be included in your project and once deployed it works great on another machine without having to install any special software. Unfortunately there are a few caveats as well. The big issue is that it hasn't been updated since 2011 so I'm assuming the project is dead, although the website still exists. Because of the lack of updates it officially only supports Internet Explorer 6, 7, 8, 9 and FireFox 2 and 3 (But I can vouch for it working fine in IE 10 + 11 on Windows 7+8). The syntax is a bit more wonky than Teleriks but it should be easy enough to use for what you need it for.
The last option that I can recommend is to use the HttpWebRequest and HttpWebResponse classes built into DotNet and do your scraping manually. I will admit that I still use this approach on occasion. It basically just brings back the source code for a certain url and you have to use string manipulation (or regex if you are good at that) to pull out the information you need. The upside to this one is that it has the least overhead as it requires no extra dlls or installs to work, and it's very fast. I will provide you a sample function that I have used and reused in a number of projects to pull data from the web:
Private Function GetMethod(ByVal sPage As String) As String
Dim req As HttpWebRequest
Dim resp As HttpWebResponse
Dim stw As StreamReader
Dim sReturnString As String = ""
Dim sUserAgent As String = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)"
Try
req = HttpWebRequest.Create(sPage)
req.Method = "GET"
req.AllowAutoRedirect = False
req.UserAgent = sUserAgent
req.Accept = "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
req.Headers.Add("Accept-Language", "en-us,en;q=0.5")
req.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7")
req.Headers.Add("Keep-Alive", "300")
req.KeepAlive = True
resp = req.GetResponse ' Get the response from the server
If req.HaveResponse Then
resp = req.GetResponse ' Get the response from the server
stw = New StreamReader(resp.GetResponseStream)
sReturnString = stw.ReadToEnd() ' Save the source code of the url into a string
Else
MessageBox.Show("No response received from host " & sPage, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
End If
Catch exc As WebException
MessageBox.Show("Network Error: " & exc.Message.ToString & " Status Code: " & exc.Status.ToString & " from " & sPage, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error)
End Try
Return sReturnString
End Function
Just a note about the sUserAgent variable: You can change this to any user agent you want to emulate. In this case I am using Mozilla Firefox 5 which is obviously super old but I have been using this function for years and haven't needed to update it.