Super slow HttpWebRequest - vb.net

Question

I'm making a website scraper for a project I'm doing. I've gotten everything to work great, however, loading the actual page takes F-O-R-E-V-E-R. You can see the page it's loading here:
MCServerList.Net
Here is the code I am using:
private CONST REQUESTURL as string = "http://www.MCServerList.net/?page="
private chunkId as int32 = 1
Dim req As HttpWebRequest = WebRequest.Create(REQUESTURL & chunkId)
Dim res As HttpWebResponse = req.GetResponse()
Dim Stream As Stream = res.GetResponseStream()
I then use "Stream" and load it through the HTMLAgilityPack found free online. It loads the page quickly, however, the initial request usually takes ~20-30 seconds.
Any help would be appreciated!

Answer 1

I just ran the following code and ignoring the first initial compile I average about 3.3 seconds for GetResponse() and 0.2 more seconds for Load(). Are you on a fast connection? Are you sure this is where the bottleneck is?
Option Explicit On
Option Strict On
Imports System.Net
Public Class Form1
Private Const REQUESTURL As String = "http://www.MCServerList.net/?page="
Private chunkId As Int32 = 1
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Dim ST As New System.Diagnostics.Stopwatch()
ST.Start()
Dim req = WebRequest.Create(REQUESTURL & chunkId)
Dim res = req.GetResponse()
Trace.WriteLine(String.Format("GetResponse() : {0}", ST.Elapsed))
Using Stream As System.IO.Stream = res.GetResponseStream()
Dim X As New HtmlAgilityPack.HtmlDocument()
X.Load(Stream)
End Using
Trace.WriteLine(String.Format("Load() : {0}", ST.Elapsed))
ST.Stop()
End Sub
End Class