Phasing website directory/ folder name vb.net - vb.net

Question

I have a website with folders i want vb.net to phase (check) for existance of these. Obviously if Directory exists dosnt work in this scenario so I came up with the following code.
Dim PhaseUri As New System.Uri(http://www.example.com/My_Folder)
Dim PhaseURL As String = PhaseUri.Host.ToString
If My.Computer.Network.IsAvailable AndAlso My.Computer.Network.Ping(PhaseURL, 1000) Then
Else
End If
The code checks if www.example.com exists by extracting it from the entire (http://www.example.com/My_Folder), its as close as I can get, however if www.example.com exists but My_Folder dosnt then I have a problem. Basicaly my above code gets me as close as possible and then makes a short leap of faith in assuming the rest will also exist.
How can I phase/ validate/ check for the existance of the entier http://www.example.com/My_Folder using less than 100,000 lines and as minimal complex code as possible?

Answer 1

It only works if your 'folder' is a webpage or contains a default document.
Works for a page, not folder
public function IsFolderOnline(url as string) as boolean
try
dim wc as new system.net.webclient()
wc.downloadstring(url)
return true
catch ex as exception
return false
end try
end function
use it like this:
dim IsItOnline as boolean = IsFolderOnline("http://example.org/folder")
it is not possible to get all pages within a folder
you must check every page for href elements to find new urls. It will only find urls that have a weblink.
Webclient.Downloadstring() downloads the html of a page.
use this code to retrieve all urls from the page
' Finding urls in the page
Dim FoundUrls As New List(Of String)
Dim strRegex As String = "<a[^<>]*>"
Dim myRegex As New Text.RegularExpressions.Regex(strRegex, System.Text.RegularExpressions.RegexOptions.None)
For Each myMatch As Text.RegularExpressions.Match In myRegex.Matches(html)
If myMatch.Success Then
Dim strRegex2 As String = "href=""[^""]*"""
Dim myRegex2 As New Text.RegularExpressions.Regex(strRegex, System.Text.RegularExpressions.RegexOptions.None)
For Each myMatch2 As Text.RegularExpressions.Match In myRegex.Matches(myMatch.Value)
If myMatch2.Success Then
Dim value As String = Split(Split(myMatch2.Value, """", 2).Last, """", 2).First
If value.ToLower.StartsWith("http://" & DomainName.ToLower) Then
FoundUrls.Add(value)
End If
If value.ToLower.StartsWith("https://" & DomainName.ToLower) Then
FoundUrls.Add(value)
End If
If value.ToLower.StartsWith("/") Then
FoundUrls.Add(Protocol & DomainName & value)
Else
If Not value.Contains(":") Then
FoundUrls.Add(Protocol & DomainName & "/" & value)
End If
End If
End If
Next
End If
Next
you might want to remove links that contain files
'Remove unsupported urls
Dim UnsupportedExtensions() As String = {".css", ".ico", ".xml", ".axd", ".js", ".zip", ".rar", ".7z", ".tar", ".jar", ".pdf", ".jpg", ".jpeg", ".png", ".bmp", ".mp4", ".mov", ".mpeg", ".gif"}
For Each url In FoundUrls.ToList
For Each UnsupportedExtension In UnsupportedExtensions
If url.ToLower.EndsWith(UnsupportedExtension) Then
FoundUrls.Remove(url)
Exit For
End If
Next
Next