Search column for urls, save webpages as individual text files - vba

Question

I have code here that works for a url that is hard coded, and it only works for one url and one text file.
Sub saveUrl_Test()
Dim FileName As String
Dim FSO As Object
Dim ieApp As Object
Dim Txt As String
Dim TxtFile As Object
Dim URL As String
URL = "www.bing.com"
FileName = "C:\mallet\bing.com.txt"
Set FSO = CreateObject("Scripting.FileSystemObject")
Set TxtFile = FSO.OpenTextFile(FileName, 2, True, -1)
Set ieApp = CreateObject("InternetExplorer.Application")
ieApp.Visible = True
ieApp.Navigate URL
While ieApp.Busy Or ieApp.ReadyState <> 4
DoEvents
Wend
Txt = ieApp.Document.body.innerText
TxtFile.Write Txt
TxtFile.Close
ieApp.Quit
Set ieApp = Nothing
Set FSO = Nothing
End Sub
What I want it to do is search in column B for urls (possibly using InStr(variable, "http://") as a boolean), and then save each webpage as an individual text file. Would there be a way to name the text files using part of the URL strings? Also, is there a way for the webpage not to open, but still save as a text file? Opening the webpages wastes a lot of time.
I created this additional sub based on #MikeD's suggestion, but I get the wend without while error.
Sub url_Test(URL As String, FileName As String)
Dim FSO As Object
Dim ieApp As Object
Dim Txt As String
Dim TxtFile As Object
Set FSO = CreateObject("Scripting.FileSystemObject")
Set TxtFile = FSO.OpenTextFile(FileName, 2, True, -1)
Set ieApp = CreateObject("InternetExplorer.Application")
ieApp.Visible = True
ieApp.Navigate URL
While ieApp.Busy Or ieApp.ReadyState <> 4
DoEvents
Wend
Txt = ieApp.Document.body.innerText
TxtFile.Write Txt
TxtFile.Close
ieApp.Quit
Set ieApp = Nothing
Set FSO = Nothing
End Sub
Sub LoopOverB()
Dim myRow As Long
myRow = 10
While Cells(myRow, 2).Value <> ""
If InStr(1, Cells(myRow, 2).Value, "http:\\", vbTextCompare) Then Call url_Test(Cells(myRow, 2).Value, "C:\mallet\test\" & Cells(myRow, 1).Value & ".txt")
myRow = myRow + 1
Wend
End Sub

Answer 1

First you could parameterize the sub
Sub saveUrl_param(URL as String, FileName as String)
....
End Sub
and remove the Dim and assignment statements for URL and FileName
Secondly you write another Sub which loops through non-empty cells in column B, retrieving values and conditionally calling the saveUrl_param() routine.
example:
Sub LoopOverB()
Dim C As Range
For Each C In Intersect(ActiveSheet.Columns("B"), ActiveSheet.UsedRange).SpecialCells(xlCellTypeConstants)
' If C = .... Then ' note: URL in [B], filename in [C]
' saveUrl_param(C, C(1,2))
' End If
Next C
End Sub
and no - you can't do it without opening the Web page; you somehow have to get the page from the server (or the proxy). This is done by
ieApp.Navigate URL
and the following While ... Wend construct waits until the page is fully loaded into the browser object.
To speed up things you could skip
ieApp.Visible = True
once you have confidence that your Sub is working correctly, and you could move
Dim ieApp As Object ' I would prefer As SHDocVw.InternetExplorer .... don't like late binding
Set ieApp = CreateObject("InternetExplorer.Application")
to the calling sub and hand over the ieApp object to the subroutine as a parameter in order to not open/close the browser again & again.