Automate download data from web site using vb.net - vb.net

Question

I have an assignment, I want to download the data from some specific web site, I checked the topic of httpwebrequest and httpwebresponse but still I m not able to understand how we can fill the text boxes on that web site and press the login button to go inside the web site, login into the web site is the first step.
I want to use VB.net for this task, if anybody can help us.
Thanks In Advance.

Answer 1

I had written an article on this using ASP.NET MVC, it should help you : Introduction to Web Scraping with HttpWebRequest using ASP.NET MVC 3
Web Scraping means parsing those pages to extract pieces of information in a structured way. It also refers to creating a programmatic interface, an API, that interacts with a site through an HTML interface meant for humans.
Please dont think that your program will type in username and password in to thier respective textboxes and then the program will press the login button for you to go inside the website.
Web Scraping does not happen in this manner.
You need to study how the website POST the Login username and password and then using your code you need to do the same thing to get the cookie. At each step, rather on each page you ll need to see how the website works using firebug or chrome dev tool and then send the POST or GET data accordingly to get what you want.
Below is what I had written to scrape data from my WordPress website, I have added comments wherever applicable to make the code easier to read.
Below are the constant that you need to define, ‘UserName’ and ‘Pwd’ are the login details to my WordPress account, ‘Url’ stand for the login page url and ‘ProfileUrl’ is the address of the page where the profile details are shown.
const string Url = "http://yassershaikh.com/wp-login.php";
const string UserName = "guest";
const string Pwd = ".netrocks!!"; // n this not my real pwd :P
const string ProfileUrl = "http://yassershaikh.com/wp-admin/profile.php";
public ActionResult Index()
{
string postData = Crawler.PreparePostData(UserName, Pwd, Url);
byte[] data = Crawler.GetEncodedData(postData);
string cookieValue = Crawler.GetCookie(Url, data);
var model = Crawler.GetUserProfile(ProfileUrl, cookieValue);
return View(model);
}
I had created a static class called “Crawler”, here’s the code for it.
// preparing post data
public static string PreparePostData(string userName, string pwd, string url)
{
var postData = new StringBuilder();
postData.Append("log=" + userName);
postData.Append("&");
postData.Append("pwd=" + pwd);
postData.Append("&");
postData.Append("wp-submit=Log+In");
postData.Append("&");
postData.Append("redirect_to=" + url);
postData.Append("&");
postData.Append("testcookie=1");
return postData.ToString();
}
public static byte[] GetEncodedData(string postData)
{
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
return data;
}
public static string GetCookie(string url, byte[] data)
{
var webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2";
webRequest.AllowAutoRedirect = false;
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data, 0, data.Length);
requestStream.Close();
var webResponse = (HttpWebResponse)webRequest.GetResponse();
string cookievalue = string.Empty;
if (webResponse.Headers != null && webResponse.Headers["Set-Cookie"] != null)
{
cookievalue = webResponse.Headers["Set-Cookie"];
// Modify CookieValue
cookievalue = GenerateActualCookieValue(cookievalue);
}
return cookievalue;
}
public static string GenerateActualCookieValue(string cookievalue)
{
var seperators = new char[] { ';', ',' };
var oldCookieValues = cookievalue.Split(seperators);
string newCookie = oldCookieValues[2] + ";" + oldCookieValues[0] + ";" + oldCookieValues[8] + ";" + "wp-settings-time-2=1345705901";
return newCookie;
}
public static List<string> GetUserProfile(string profileUrl, string cookieValue)
{
var webRequest = (HttpWebRequest)WebRequest.Create(profileUrl);
webRequest.Method = "GET";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2";
webRequest.AllowAutoRedirect = false;
webRequest.Headers.Add("Cookie", cookieValue);
var responseCsv = (HttpWebResponse)webRequest.GetResponse();
Stream response = responseCsv.GetResponseStream();
var htmlDocument = new HtmlDocument();
htmlDocument.Load(response);
var responseList = new List<string>();
// reading all input tags in the page
var inputs = htmlDocument.DocumentNode.Descendants("input");
foreach (var input in inputs)
{
if (input.Attributes != null)
{
if (input.Attributes["id"] != null && input.Attributes["value"] != null)
{
responseList.Add(input.Attributes["id"].Value + " = " + input.Attributes["value"].Value);
}
}
}
return responseList;
}
Hope this helps.