Excel VBA - Copy table data from website and put into excel? - vba

I am trying to copy a table from a website and put this into excel.
My table can be identified by the elementID "VisibleReportContentctl32".
For some reason, my code produces this error:
'object required'
on this line: Range("A1").Text = dat
Here's my full code:
Option Explicit
Public Sub GetSSRSData()
Dim IE As Object: Set IE = New InternetExplorerMedium
Dim TR_Elements As Object
Dim TR As Object ' Table Row
Dim TD_Elements As Object
Dim TD As Object ' Table Data
Dim RowNumb As Integer
Dim Columns As Integer
Dim ColumnNumb As Integer
Dim x As Integer
Dim dat As String
With IE
.Visible = True
.Navigate ("http://gbrlon02-sql-17/Reports/Pages/Report.aspx?ItemPath=%2fCutlass+Reports%2fManagement+Reporting%2fForwardOrdersSticksOnly+-+ForecastVariance+(Monthly+Report)")
Do While IE.Busy Or IE.ReadyState <> 4
DoEvents
Loop
If IsObject(.Document.getElementById("VisibleReportContentctl32_ctl09")) Then
dat = IE.Document.getElementById("VisibleReportContentctl32_ctl09").innerHTML
Range("A1").Text = dat
Else
MsgBox "doesn't"
End If
End With
End Sub
please can someone show me where I am going wrong?

Use [Power Query][Table from web] is simple and stable

That URL doesn't work for me. Your code should look something like this...
Option Explicit
Sub gethtmltable()
Dim objWeb As QueryTable
Dim sWebTable As String
'You have to count down the tables on the URL listed in your query
'This example shows how to retrieve the 2nd table from the web page.
sWebTable = 2
'Sets the url to run the query and the destination in the excel file
'You can change both to suit your needs
Set objWeb = ActiveSheet.QueryTables.Add( _
Connection:="URL;http://www.vbaexpress.com/kb/default.php", _
Destination:=Range("A1"))
With objWeb
.WebSelectionType = xlSpecifiedTables
.WebTables = sWebTable
.Refresh BackgroundQuery:=False
.SaveData = True
End With
Set objWeb = Nothing
End Sub
You just need to identify which specific table you want to import data from.

Related

Excel VBA-Run-time error 438. Object doesn't support this property or method. When trying to copy website table to excel

I am trying to extract table from an internal website to excel by giving some input parameters. Everything works until it refreshes the website data with my inputs. The part I get the run-time error 438 is marked (For r = 1 To elemCollection.Rows.Length - 1). I also tried to load the data from website to excel using web query and the table wasn't showing up on my excel spreadsheet. "It gives the following error-This page might not function correctly because either your browser does not support scripts or active scripting is disabled. Your browser does not support scripts or has been configured not to allow scripts. The report viewer web control http handler has not been registered in the application's web config file."
Wondering if this has anything to do with permissions.
VBA code below:
Option Explicit
Sub Macro1()
Dim IE As Object, obj As Object
Dim StartDate As Object
Dim EndDate As Object
Dim myState As String
Dim r As Integer, c As Integer, t As Integer
Dim elemCollection As Object, curHTMLRow As Object
Dim wb As Workbook
Dim ws As Worksheet
Dim objCollection As Object
Dim objElement As Object
Dim i As Long
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate ("http://internalwebsite_SSRSReport")
' we ensure that the web page downloads completely before we fill the form automatically
While IE.ReadyState <> 4: DoEvents: Wend
IE.Document.All.Item("ctl31_ctl03_txtValue").InnerText = Format("7/1/2016", "m/d/yyyy")
IE.Document.All.Item("ctl31_ctl05_txtValue").InnerText = Format("7/31/2016", "m/d/yyyy")
Wait 2
IE.Document.getElementsByName("ctl31_ctl04_divDropDown").Item.Click
Wait 2
' accessing the button
IE.Document.getElementsByName("ctl31_ctl04_ctl00").Item.Click
Wait 2
' again ensuring that the web page loads completely before we start scraping data
While IE.busy: DoEvents: Wend
Wait 2
'Clearing any unnecessary or old data in Sheet1
ThisWorkbook.Sheets("Sheet1").Activate
Range("A1:K500").ClearContents
Set elemCollection = IE.Document.getelementbyId("ctl31_ctl09_ReportArea")
'error here
For r = 1 To elemCollection.Rows.Length - 1
Set curHTMLRow = elemCollection.Rows(r)
For c = 0 To curHTMLRow.Cells.Length - 1
Cells(r + 1, c + 1) = curHTMLRow.Cells(c).InnerText
Next
Next
' cleaning up memory
IE.Quit
Set IE = Nothing
End Sub
Private Sub Wait(ByVal nSec As Long)
nSec = nSec + Timer
While nSec > Timer
DoEvents
Wend
End Sub
Below is some code that should be able to grab the data from the HTML table from the SSRS report and extract it to Excel.
Basically the code will iterate through all the TRs and TDs in the Table Element, and output the InnerText to Excel. If you are moving a lot of data over, please consider writing to an array, then doing the write all at once by setting to an equally sized range object.
I also cleaned up the code, mostly removing variables that were not referenced and reduced some of the lines by combining some statements together
Option Explicit
Public Sub GetSSRSData()
On Error GoTo errhand:
Application.ScreenUpdating = False
Dim IE As Object: Set IE = CreateObject("InternetExplorer.Application")
Dim TR_Elements As Object
Dim TR As Object ' Table Row
Dim TD_Elements As Object
Dim TD As Object ' Table Data
Dim RowNumb As Integer
Dim Columns As Integer
Dim ColumnNumb As Integer
With IE
.Visible = True
.Navigate ("http://internalwebsite_SSRSReport")
While .ReadyState <> 4: DoEvents: Wend ' Wait for page load
'Fill the form out with dates
.Document.All.Item("ctl31_ctl03_txtValue").InnerText = Format("7/1/2016", "m/d/yyyy")
.Document.All.Item("ctl31_ctl05_txtValue").InnerText = Format("7/31/2016", "m/d/yyyy")
Wait 2
'Click the DropDown
.Document.getElementsByName("ctl31_ctl04_divDropDown").Item.Click
Wait 2
' Click the other button
.Document.getElementsByName("ctl31_ctl04_ctl00").Item.Click
End With
Wait 2
While IE.busy: DoEvents: Wend ' Wait for page load
Wait 2
'Clearing any unnecessary or old data in Sheet1
Sheets("Sheet1").Range("A1:K500").ClearContents
Set TR_Elements = IE.Document.getelementbyId("ctl31_ctl09_ReportArea").getElementsByTagName("tr")
RowNumb = 1
ColumnNumb = 1
'Tables usually consists of TR (Table Rows), and -
'TD (Table Data)
For Each TR In TR_Elements
Set TD_Elements = TR.getElementsByTagName("td")
ColumnNumb = 1
For Each TD In TD_Elements
'Consider using an array to save the values to memory if there is going
'to be a lot of data to be moved over
ActiveSheet.Cells(RowNumb, ColumnNumb).Value = TD.InnerText
ColumnNumb = ColumnNumb + 1
Next
RowNumb = RowNumb + 1
Next
' cleaning up memory
IE.Quit
Set IE = Nothing
Set TD_Elements = Nothing
Set TR_Elements = Nothing
Set TD = Nothing
Set TR = Nothing
Application.ScreenUpdating = True
errhand:
Application.ScreenUpdating = True
End Sub
Private Sub Wait(ByVal nSec As Long)
nSec = nSec + Timer
While nSec > Timer
DoEvents
Wend
End Sub

Export query to Excel and put data into table MS Access 2013 VBA

I have already been able to export a query from MS Access to an Excel workbook and autoformat the column widths and other settings, but I cannot find out how to put this data into a table. I found the command to create a table which is this:
Sheet1.ListObjects.Add(xlSrcRange, Range("A1:D10"), , xlYes).Name = "myTable1"
but that is hardcoding the size of the table. Since I am exporting multiple queries, I want to have a modular function which will take queries of different column/row lengths and create tables for all of them without having to manually type the size. Here is some of my code:
Private Sub dumpQueries(path As String)
Dim obj As AccessObject, dB As Object
Set dB = Application.CurrentData
For Each obj In dB.AllQueries
testBool = InStr(obj.name, "Sys")
If testBool <> True Then
If obj.name = "example1" Or obj.name = "example2" Then
DoCmd.TransferSpreadsheet acExport, acSpreadsheetTypeExcel12Xml, obj.name, path, True, editWorksheetName(obj.name)
End If
End If
Next obj
End Sub
Private Sub formatFile(path As String)
Dim Date1 As Date, strReportAddress As String
Dim objActiveWkb As Object, appExcel As Object
Dim sht As Worksheet
Dim LastRow As Long
Dim LastColumn As Long
Dim StartCell As Range
Set appExcel = CreateObject("Excel.Application")
appExcel.Visible = False
appExcel.Application.Workbooks.Open (path)
Set objActiveWkb = appExcel.Application.ActiveWorkbook
With objActiveWkb
Dim i As Integer
For i = 1 To .Worksheets.count
.Worksheets(i).Select
Set sht = Worksheets(i)
Set StartCell = Range("A1")
.Worksheets(i).Cells.Select
.Worksheets(i).Cells.EntireColumn.AutoFit
.Worksheets(i).UsedRange
LastRow = StartCell.SpecialCells(xlCellTypeLastCell).Row
LastColumn = StartCell.SpecialCells(xlCellTypeLastCell).Column
sht.Range(StartCell, sht.Cells(LastRow, LastColumn)).Select
Next
End With
appExcel.ActiveWindow.TabRatio = 0.7
objActiveWkb.Close savechanges:=True
appExcel.Application.Quit
Set objActiveWkb = Nothing: Set appExcel = Nothing
End Sub
There is a lot more code but this is the relevent stuff. This is where I create the excel files and format them. Any idea how to put this data directly into a table?
Update: I fixed all the errors I was getting but it still doesn't create a table with all the data. I edited my code above to be completely updated.
Fixed this problem, but new one came up. Please go to VBA Run-time error 1004: Method Range of object _Global failed when trying to create tables in Excel 2013 if you can help.
Consider using QueryTables and specify the upper left corner destination and specific query. Below is Excel VBA code where you import via ODBC from external Access database:
Dim constr As String
constr = "ODBC;DRIVER=Microsoft Access Driver (*.mdb, *.accdb);" _
& "DBQ=C:\Path\To\Database\File.accdb;"
With ActiveSheet.ListObjects.Add(SourceType:=0, _
Source:=constr, _
Destination:=Range("$A$1")).QueryTable
.CommandText = "SELECT * FROM [Table]"
.ListObject.DisplayName = "TableName"
.Refresh BackgroundQuery:=False
End With
Can you try using an ODBC Query? Check out the link below and see if that gets you what you want.
http://translate.google.pl/translate?js=n&prev=_t&hl=pl&ie=UTF-8&layout=2&eotf=1&sl=pl&tl=en&u=http%3A%2F%2Fafin.net%2FKsiazkaSQLwExcelu%2FGraficznyEdytorZapytanSqlNaPrzykladzieMsQuery.htm

Debugging a QueryTables.Add script

Sub FindData()
Dim accountNumber As Range
Set accountNumber = Range(Range("A2"), Range("A2").End(xlDown))
Dim dataSet As QueryTable
For Each Value In accountNumber
Set dataSet = .QueryTables.Add( _
Connection:="URL;http://www.prad.org/CamaDisplay.aspx?OutputMode=Display&SearchType=RealEstate&ParcelID=" & Value, _
Destination:=ThisWorkbook.Worksheets(2).Range("A1"))
Next Value
With dataSet
.RefreshOnFileOpen = False
.WebFormatting = xlWebFormattingNone
.BackgroundQuery = True
.WebSelectionType = xlSpecifiedTables
.WebTables = "3"
End With
With Application
dataSet.Refresh BackgroundQuery:=False
End With
End Sub
The ultimate goal here is to pull data from the URL and drop it into Worksheet(2). The values in accountNumber go at the end of the URL for each page to draw data from.
This is my first VBA script, and right off the bat, it's giving me an error on Sub FindData()
I have the table of accountNumbers. The URL for one account is the given URL with an accountNumber after the final =. I am trying to iterate through one webpage per accountNumber and extract from each.
Set dataSet = ActiveSheet.QueryTables.Add( _
Connection:="URL;http://www.prad.org/CamaDisplay.aspx?OutputMode=Display&SearchType=RealEstate&ParcelID=" & Value, _
Destination:=ThisWorkbook.Worksheets(2).Range("A1"))
QueryTables needs to be properly referenced. You can use a sheet qualifier like :
Sheets("yourname").QueryTables or something.
You can remove the dot too...
Look into my code and see if this helps. I added a lot of comments to help you understand better the way the whole thing works.
Option Explicit
Sub FindData()
Const strURL As String = "URL;http://www.prad.org/CamaDisplay.aspx?OutputMode=Display&SearchType=RealEstate&ParcelID="
Dim shActive As Worksheet
Dim shDestination As Worksheet
Dim oQuery As QueryTable
Dim rAccounts As Range
Dim rAccount As Range
'Initialize the variables
Set shActive = ActiveSheet
' Note the "." in front of the ranges. That's how you use "With"
With shActive
Set rAccounts = .Range(.Range("A2"), .Range("A2").End(xlDown))
End With
' Remove any old query otherwise they will pile up and slow down
' your workbook
Call RemoveSheetQueries(shActive)
' Loop through the accounts and add the queries
For Each rAccount In rAccounts
Set oQuery = Nothing
Set oQuery = shActive.QueryTables.Add(Connection:=strURL & rAccount.Value, _
Destination:=shActive.Range("A1"))
' Set the properties of the new query and eventually run it.
With oQuery
.RefreshOnFileOpen = False
.WebFormatting = xlWebFormattingNone
.BackgroundQuery = True
.WebSelectionType = xlSpecifiedTables
.WebTables = "3"
' This last line will actually get the data
.Refresh BackgroundQuery:=False
End With
Next rAccount
End Sub
' Procedure to remove all old Queries
Sub RemoveSheetQueries(ByRef shToProcess As Worksheet)
Dim lTotal As Long
Dim i As Long
lTotal = shToProcess.QueryTables.Count
For i = lTotal To 1 Step -1
shToProcess.QueryTables(i).Delete
Next i
End Sub
I hope it helps :)

Excel VBA Macro: Scraping data from site table that spans multiple pages

Thanks in advance for the help. I'm running Windows 8.1, I have the latest IE / Chrome browsers, and the latest Excel. I'm trying to write an Excel Macro that pulls data from StackOverflow (https://stackoverflow.com/tags). Specifically, I'm trying to pull the date (that the macro is run), the tag names, the # of tags, and the brief description of what the tag is. I have it working for the first page of the table, but not for the rest (there are 1132 pages at the moment). Right now, it overwrites the data everytime I run the macro, and I'm not sure how to make it look for the next empty cell before running.. Lastly, I'm trying to make it run automatically once per week.
I'd much appreciate any help here. Problems are:
Pulling data from the web table beyond the first page
Making it scrape data to the next empty row rather than overwriting
Making the Macro run automatically once per week
Code (so far) is below. Thanks!
Enum READYSTATE
READYSTATE_UNINITIALIZED = 0
READYSTATE_LOADING = 1
READYSTATE_LOADED = 2
READYSTATE_INTERACTIVE = 3
READYSTATE_COMPLETE = 4
End Enum
Sub ImportStackOverflowData()
'to refer to the running copy of Internet Explorer
Dim ie As InternetExplorer
'to refer to the HTML document returned
Dim html As HTMLDocument
'open Internet Explorer in memory, and go to website
Set ie = New InternetExplorer
ie.Visible = False
ie.navigate "http://stackoverflow.com/tags"
'Wait until IE is done loading page
Do While ie.READYSTATE <> READYSTATE_COMPLETE
Application.StatusBar = "Trying to go to StackOverflow ..."
DoEvents
Loop
'show text of HTML document returned
Set html = ie.document
'close down IE and reset status bar
Set ie = Nothing
Application.StatusBar = ""
'clear old data out and put titles in
'Cells.Clear
'put heading across the top of row 3
Range("A3").Value = "Date Pulled"
Range("B3").Value = "Keyword"
Range("C3").Value = "# Of Tags"
'Range("C3").Value = "Asked This Week"
Range("D3").Value = "Description"
Dim TagList As IHTMLElement
Dim Tags As IHTMLElementCollection
Dim Tag As IHTMLElement
Dim RowNumber As Long
Dim TagFields As IHTMLElementCollection
Dim TagField As IHTMLElement
Dim Keyword As String
Dim NumberOfTags As String
'Dim AskedThisWeek As String
Dim TagDescription As String
'Dim QuestionFieldLinks As IHTMLElementCollection
Dim TodaysDate As Date
Set TagList = html.getElementById("tags-browser")
Set Tags = html.getElementsByClassName("tag-cell")
RowNumber = 4
For Each Tag In Tags
'if this is the tag containing the details, process it
If Tag.className = "tag-cell" Then
'get a list of all of the parts of this question,
'and loop over them
Set TagFields = Tag.all
For Each TagField In TagFields
'if this is the keyword, store it
If TagField.className = "post-tag" Then
'store the text value
Keyword = TagField.innerText
Cells(RowNumber, 2).Value = TagField.innerText
End If
If TagField.className = "item-multiplier-count" Then
'store the integer for number of tags
NumberOfTags = TagField.innerText
'NumberOfTags = Replace(NumberOfTags, "x", "")
Cells(RowNumber, 3).Value = Trim(NumberOfTags)
End If
If TagField.className = "excerpt" Then
Description = TagField.innerText
Cells(RowNumber, 4).Value = TagField.innerText
End If
TodaysDate = Format(Now, "MM/dd/yy")
Cells(RowNumber, 1).Value = TodaysDate
Next TagField
'go on to next row of worksheet
RowNumber = RowNumber + 1
End If
Next
Set html = Nothing
'do some final formatting
Range("A3").CurrentRegion.WrapText = False
Range("A3").CurrentRegion.EntireColumn.AutoFit
Range("A1:C1").EntireColumn.HorizontalAlignment = xlCenter
Range("A1:D1").Merge
Range("A1").Value = "StackOverflow Tag Trends"
Range("A1").Font.Bold = True
Application.StatusBar = ""
MsgBox "Done!"
End Sub
There's no need to scrape Stack Overflow when they make the underlying data available to you through things like the Data Explorer. Using this query in the Data Explorer should get you the results you need:
select t.TagName, t.Count, p.Body
from Tags t inner join Posts p
on t.ExcerptPostId = p.Id
order by t.count desc;
The permalink to that query is here and the "Download CSV" option which appears after the query runs is probably the easiest way to get the data into Excel. If you wanted to automate that part of things, the direct link to the CSV download of results is here
You can improve this to parse out exact elements but it loops all the pages and grabs all the tag info (everything next to a tag)
Option Explicit
Public Sub ImportStackOverflowData()
Dim ie As New InternetExplorer, html As HTMLDocument
Application.ScreenUpdating = False
With ie
.Visible = True
.navigate "https://stackoverflow.com/tags"
While .Busy Or .READYSTATE < 4: DoEvents: Wend
Set html = .document
Dim numPages As Long, i As Long, info As Object, item As Object, counter As Long
numPages = html.querySelector(".page-numbers.dots ~ a").innerText
For i = 1 To 2 ' numPages ''<==1 to 2 for testing; use to numPages
DoEvents
Set info = html.getElementById("tags_list")
For Each item In info.getElementsByClassName("grid-layout--cell tag-cell")
counter = counter + 1
Cells(counter, 1) = item.innerText
Next item
html.querySelector(".page-numbers.next").Click
While .Busy Or .READYSTATE < 4: DoEvents: Wend
Set html = .document
Next i
Application.ScreenUpdating = True
.Quit '<== Remember to quit application
End With
End Sub
I'm not making use of the DOM, but I find it very easy to get around just searching between known tags. If ever the expressions you are looking for are too common just tweak the code a bit so that it looks for a string after a string).
An example:
Public Sub ZipLookUp()
Dim URL As String, xmlHTTP As Object, html As Object, htmlResponse As String
Dim SStr As String, EStr As String, EndS As Integer, StartS As Integer
Dim Zip4Digit As String
URL = "https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=1&companyName=&address1=1642+Harmon+Street&address2=&city=Berkeley&state=CA&urbanCode=&postalCode=&zip=94703"
Set xmlHTTP = CreateObject("MSXML2.XMLHTTP")
xmlHTTP.Open "GET", URL, False
On Error GoTo NoConnect
xmlHTTP.send
On Error GoTo 0
Set html = CreateObject("htmlfile")
htmlResponse = xmlHTTP.ResponseText
If htmlResponse = Null Then
MsgBox ("Aborted Run - HTML response was null")
Application.ScreenUpdating = True
GoTo End_Prog
End If
'Searching for a string within 2 strings
SStr = "<span class=""address1 range"">" ' first string
EStr = "</span><br />" ' second string
StartS = InStr(1, htmlResponse, SStr, vbTextCompare) + Len(SStr)
EndS = InStr(StartS, htmlResponse, EStr, vbTextCompare)
Zip4Digit = Left(Mid(htmlResponse, StartS, EndS - StartS), 4)
MsgBox Zip4Digit
GoTo End_Prog
NoConnect:
If Err = -2147467259 Or Err = -2146697211 Then MsgBox "Error - No Connection": GoTo End_Prog 'MsgBox Err & ": " & Error(Err)
End_Prog:
End Sub

Get website data from Urls using VBA

I have multiple urls stored in Excel sheet. I want to Get data reside within particular div tag. For One Website it works fine
Sub Cityline()
Dim IE As Object
Set IE = CreateObject("Internetexplorer.application")
IE.Visible = True
IE.navigate "http://Someurl.com/bla/bla/bla"
Do While IE.busy
DoEvents
Loop
Do
DoEvents
Dim Doc As Object
Set Doc = IE.Document
Dim workout As String
workout = Doc.getElementsByClassName("CLASS_NAME_OF_DATA")(0).innertext
Range("A2") = workout
Loop
End Sub
I used Below code for loop Through all urls but its not working
Sub GetData()
Dim oHtm As Object: Set oHtm = CreateObject("HTMLFile")
Dim req As Object: Set req = CreateObject("msxml2.xmlhttp")
Dim oRow As Object
Dim oCell As Range
Dim url As String
Dim y As Long, x As Long
x = 1
For Each oCell In Sheets("sheet1").Range("A2:A340")
req.Open "GET", oCell.Offset(, 1).Value, False
req.send
With oHtm
.body.innerhtml = req.responsetext
With .getelementsbytagname("table")(1)
With Sheets(1)
.Cells(x, 1).Value = oCell.Offset(, -1).Value
.Cells(x, 2).Value = oCell.Value
End With
y = 3
For Each oRow In .Rows
Sheets(1).Cells(x, y).Value = oRow.Cells(1).innertext
y = y + 1
Next oRow
End With
End With
x = x + 1
Next oCell
End Sub
But its not working
can any one suggest me where i went wrong ?
I used Fetching Data from multiple URLs but it doesn't works for me.
Please guide me how to get data from all urls at a Time
I'm new to SO, so apologies to the mods if this should be in comments (I couldn't get it to fit).
I agree with Silver's comments, but I thought I'd suggest a different approach that might help. If you have URLs in a column of cells, you could create a custom VBA function that will extract the relevant data out of the HTML. Just use this function in the cells to the right of your URL to return the relevant data from the HTML. An example is this:
Public Function GetHTMLData(SiteURL As String, FieldSearch As String) As String
Dim IE As Object
Dim BodyHTML As String
Dim FieldStart As Integer
Dim FieldEnd As Integer
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Navigate SiteURL
Do While .Busy Or .ReadyState <> 4
DoEvents
Loop
BodyHTML = IIf(StrComp(.Document.Title, "Cannot find server", vbTextCompare) = 0, _
vbNullString, .Document.body.innerhtml)
FieldStart = InStr(1, BodyHTML, FieldSearch) + Len(FieldSearch) + 12
FieldEnd = InStr(FieldStart, BodyHTML, "<")
GetHTMLData = Mid(BodyHTML, FieldStart, FieldEnd - FieldStart)
.Quit
End With
Set IE = Nothing
End Function
The function above has 2 input parameters: the URL and a string that will be searched for within the HTML. It will then return a string from within the HTML, starting from 12 characters after the searched parameter and ending at the following '<' within the HTML.
Hope that helps.