VBA Web page scrape finishes before page loads - vba

I'm doing a web scrape in VBA (see code below) to btc-e.com to fetch prices of some Cryptocurrency. When to it manually by going to data tab and then clicking on from web It works fine, but When I do it in the Macro I only get back "please wait..."
The page displays "please wait..." as it loads and the macro assumes that is the entire page.
I have been looking for a way to make the macro wait for the full page load and cant find anything.
Any help would be appreciated.
Thanks
With ActiveSheet.QueryTables.Add(connection:="URL;https://btc-e.com", _
Destination:=Range("$A$1"))
.Name = "btc-e"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False ' was true
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False ' True ' was false
.WebDisableRedirections = False 'True ' was false
.Refresh BackgroundQuery:=False
End With

You have to choose a specific table or area of the page you want scraped, otherwise it won't work. The reason is that you are automatically forwarded from the page you are trying to scrape to the page you can actually scrape.
When I chose to scrape the Sell Orders, this is the code I got from the macro recorder:
With ActiveSheet.QueryTables.Add(Connection:="URL;https://btc-e.com", _
Destination:=Range("$B$2"))
.Name = "btc-e"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "3"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
As you can see it takes the attribute ".WebTables" which chooses a specific portion of the site. You can choose the portion you want by activating the macro recorder, scraping it through the normal way, choosing the area you want and then looking at the value of WebTables in the resulting code.
Hope this helps!

Related

Web Scraping - Container Tracking from Searates Website to Excel

As lacking of essential IT knowledge, I am looking for your kind help to make codes for getting data (final destination & ETA estimated time of arrival) from website tracking container numbers SEARATES (https://www.searates.com/container/tracking/) to Excel.
I have tried method of using GetElementbyID using IE Internet Explorer but this website browser is seems obsolete and useless.
This SERATES website offer API information but it is too complicated for me to understand.
I am enclosing container tracking list for your testing.
https://drive.google.com/file/d/1E0tuA4pYMDPZMgwjQC8dDs6w6BxqVdXK/view?usp=sharing
Hope someone could give me support on this project which is important for my job of logistics.
Thanks in advance.
How about recording a Macro, for starters.
Sub Macro1()
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;https://drive.google.com/file/d/1E0tuA4pYMDPZMgwjQC8dDs6w6BxqVdXK/view", _
Destination:=Range("$A$1"))
.CommandType = 0
.Name = "view"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
Range("A1").Select
Application.Left = 201.25
Application.Top = 93.25
End Sub
Notice, you have to be signed in. I don't have login creds for this link.

Type mismatch macro for external data

Hello need some help with the below code but an explanation in what I am needing might help you.
I am creating a macro that takes external data( FX rates) and places them down in a specified range on sheet1. sheet2 has just a little sum conversion using the FX rate from sheet1 to give GBP rate. This will live conversions when people fill in expenses for me on a project.
My issue is making the macro select the correct sheet.
I have come up with the below however. i'm now getting a runtime error 13.type mismatch.
before it just selected the activesheet and not the sheet I wanted.
hope you guys can help.
All new to VBA and i cannot find right answer when researching.
so any help will be greatly appreciated.
regards.
Ross
Sub fxRate()
'
' fxRate Macro
'
'
Set activateFXSheet = Sheet1.Activate
Set FXSheet = activateFXSheet.QueryTables.Add(Connection:= _
"URL;http://www.x-rates.com/table/?from=USD&amount=1", Destination:=Range( _
"$C$4:$E$14"))
With FXSheet
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "1"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub
Here you go for the ActiveSheet:
Sub fxRate()
Sheet1.Activate
Set activateFXSheet = ActiveSheet
End Sub
But using ActiveSheet and etc is not a good idea in VBA:
How to avoid using Select in Excel VBA
You don't need to select or activate a worksheet in order to reference or modify/update it.
Sub fxRate()
With Sheet1.QueryTables.Add(Connection:="URL;http://www.x-rates.com/table/?from=USD&amount=1", Destination:=Range("$C$4:$E$14"))
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "1"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub

Extracting tabular form data from multiple webpages into an excel using VBA macros..!

Extracting tabular form data from multiple webpages into an excel using VBA macros..!! Currently iam using below link but i could able to only one webpage in the code..i have list of ulr's to get data from...and it has to come in vertical..!! please suggest me.. :)
Sub INDEXdata()
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;http://recorder.maricopa.gov/recdocdata/GetRecDataDetail.aspx?rec=19770000007" _
, Destination:=Range("$A$1"))
.Name = "rec=19770000006"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "2,3"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub
OK, I don't know how much programming background you have, and I also don't know what parts of the code you posted are specific to that one source url and target location.
But, something like this might work. I made the assumption that the url, the destination, and the name would change for each page you wanted to pull from.
What I did was take the part of the code that looked like it would be true for all of the source pages and destinations, and put that in its own parameterized subroutine. The original routine IndexData just specifies the URL and the destination, and the name, for each copy operation.
Sub IndexData()
GetData("http://recorder.maricopa.gov/recdocdata/GetRecDataDetail.aspx?rec=19770000007" , _
"$A$1", _
"rec=19770000006")
GetData("http://somewhereelse.com/somedata.aspx?rec=12345", _
"$A$2", _
"rec=12345")
GetData("http://anotherurl.com/etc", _
"$A$3", _
"something")
End
Sub GetData(url as string, destination as string, name as string)
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;" & url , Destination:=Range(destination))
.Name = name
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlSpecifiedTables
.WebFormatting = xlWebFormattingNone
.WebTables = "2,3"
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub

Fetch Website data into Excel

The most recent answer to Fetch Website data into Excel was:
Sub FetchData()
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;http://www.zillow.com/homes/comps/67083361_zpid", Destination:=Range( _
"$A$1"))
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub
It works well for my project as well except the URL of my interest is:
URL;http://reservations.example.com/cp/reports/daily_usage.php?type=date_reservations&date=06%2F03%2F2015&date_end=&vessel=&club=6&location=135", Destination:=Range( _
"$A$1"))
I have used a fake URL above for privacy reasons, but as you can see, the URL has a date code within. In this case it is June 3rd, 2015 as designated by 06%2F03%2F2015.
Using VBA, how can I make the date within the URL a variable, defined within a work sheet?
With the Format() function most probably. A wild guess would be
Format(Range("$A$1").Value2, "mm""%2F""dd""%2F""yyyy")
You can also reconstruct it by yourself using the Day(), the Month() and the Year() functions and some playing around with IIf(Len(x)=2,x,0&x).
My sample gave "06%2F03%2F2014" for 2014-06-03.

Extract Data from a Web Page - using VBA

Using VBA, I need to extract data from webpage http://emops.tse.com.tw/t21/sii/t21sc03_2011_9_e.htm
I am able to fetch all the data using following code:
With ActiveSheet.QueryTables.Add(Connection:="URL;http://emops.tse.com.tw/t21/sii/t21sc03_2012_2_e.htm", Destination:=Range("$A$1"))
.Name = "67083361_zpid"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
But the problem is I don't want data from whole page. I want data from the table where Industry name is Electron (It is the last table in this case)
Any trick for the same please?
Change:
.WebSelectionType = xlEntirePage to .WebSelectionType = xlSpecifiedTables
Add:
.WebTables = "2" below .WebFormatting = xlWebFormattingNone
'You will have to use trial and error with the "2" to find the exact table you are wanting to grab