I have a worksheet Data which web scrapes a data based on a dynamic link. There is another PivotTable with pivot tables based on the Data worksheet.
Data worksheet uses the following macro and clears the contents of the cells before web scraping new updated data. This data is updated every 1 minute.
I have the following code which will refresh the pivot tables on data update.
ThisWorkbook.Worksheets("PivotTable").PivotTables("PivotTable1").RefreshTable
Since the data takes about 20 seconds to complete updating, there is no data (as the cell contents are cleared first) for the pivot table to refresh. So, I get an error.
Data uses the following code to update data:
With ThisWorkbook.Worksheets("Data").QueryTables.Add(Connection:= _
"<URL redacted>", Destination:=ThisWorkbook.Worksheets("Data").Range("$A$1"))
.Name = "DataPull"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=True ' .Delete
End With
I have tried updating the .RefreshStyle = xlInsertDeleteCells to .RefreshStyle = xlOverwriteCells. But it overwrites the cells until the end of the rows of the new data. If new data (number of rows) is less than old data's rows, then the old data rows at the end are not deleted.
I only want the data from the latest update to be kept.
How do I auto refresh the pivot tables based on above conditions?
Just set .BackgroundQuery = False so that your query will be performed synchronously (meaning, it will wait for the data to be loaded before doing the pivot refresh).
Try using a do loop while to wait for the scraping to complete.
Do
Err.Clear
On Error Resume Next
Debug.Print Err.Number
ThisWorkbook.Worksheets("PivotTable").PivotTables("PivotTable1").RefreshTable
Debug.Print Err.Number
Loop While Err.Number > 0
Related
I had some exceptional requirement. That is i have set of value which will come in the drop down in an Excel cell. While changing the value it will hit the server and fetch data according the drop down selected in the excel.
Finally i have done with some simple steps. Please find it following.
Step 1:- Create your drop down list
Go to Data -> Data validation
Now you have created a drop down in F2 cell as follows:
Now on change we have to create a macro which will fetch data from server:
Step 2 :-
Now you have to click on View Code which will open a code window. There you have to paste the following code.
Private Sub Worksheet_Change(ByVal Target As Range)
'MsgBox "You just changed " & Target.Address
If Target.Address = "$F$2" Then
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;https://www.google.com/", Destination:= _
Range("A1"))
.Name = "MR07"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = True
.WebDisableRedirections = True
.Refresh BackgroundQuery:=False
End With
End If
End Sub
In the above code you can replace your own URL instead of https://www.google.com/
A1 is the starting row of output, which is coming from server. $F$2 is validating the changes coming from drop down. if you not put this condition for all cell changes it will call this function which is not expected.
I have a connection that web scrapes based on a dynamic link. So, I cannot set a fixed connection. The following macro creates the connection and then updates the worksheet.
With ThisWorkbook.Worksheets("Data").QueryTables.Add(Connection:= _
"<URL redacted>", Destination:=ThisWorkbook.Worksheets("Data").Range("$A$1"))
.Name = "DataPull"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = False
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
.Delete
End With
This macro runs every 1 minute to update the data. So, this creates a new connection every time it runs. I do not want so many connections to exist as they won't be used anymore.
How do I delete the connection once web scraping is complete?
or Is there a way to set up a single connection that can modify itself based on a variable. The variable is the time intervals which vary based on current time.
I looked at this option
For Each qr In ThisWorkbook.Queries
qr.Delete
Next qr
But there are two other fixed connections which I don't want to delete.
The new connections that are created have the names Connection, Connection1, and so on. Is there a way to delete the connections based on name?
Let's say you want to delete all connections, except Connection1 and Connection2, try...
Dim Conn As WorkbookConnection
For Each Conn In ThisWorkbook.Connections
If Conn.Name <> "Connection1" And Conn.Name <> "Connection2" Then
Conn.Delete
End If
Next Conn
Hope this helps!
I am having a little trouble returning SQL data into an Excel table, instead of dropping the data into the cell I specify (when not using a table) it pushes the whole table to the right.
Does anyone know how I can get it to populate the table as I want to make use of splicers etc. to filter the data. I know I could import the data then turn it into a table but if there's a quicker, simpler way then I'd rather that.
Right now the code I'm using is:
With ActiveSheet.QueryTables.Add(Connection:="ODBC;DSN=<dbname>", Destination:=ActiveSheet.Range("A13"))
.CommandText = strSQL
.Name = "Query"
.FieldNames = False
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = False
.RefreshPeriod = 0
.PreserveColumnInfo = True
.Refresh BackgroundQuery:=False
End With
ActiveSheet.QueryTables(1).Delete
Many thanks in advance.
Dan
so in all my VBA macros that use SQL to query our database, I have just been using what is automatically entered when recording a macro in Excel. Is there a way to simplify it? It inserts it into a table first, but really I just want to the raw data in a spreadsheet... it doesn't need to look nice.
Sheets("Sheet1").Select
With ActiveSheet.ListObjects.Add(SourceType:=0, Source:="ODBC;DSN=AS400;" _
, Destination:=Range("$A$1")).QueryTable
.CommandText = strSQL
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.PreserveColumnInfo = True
.ListObject.DisplayName = "Table_LOCAL_ITEM"
.Refresh BackgroundQuery:=False
End With
ActiveSheet.ListObjects("Table_LOCAL_ITEM").Unlist
Cells.Select
Selection.ClearFormats
I cannot remember the exact code, but can look it up when I get to work again. But depending on which method you use to fetch the data in your SQL (e.g. ADO or DAO), I would do it something like this, if I just wanted the raw data.
If you define a variable as your recordset (for example Dim rst as recordset), VBA has a function to return all data from your recordset, something like:
Sheets("Sheet 1").Range("A1").CopyFromRecordset rst
Which will output all the data from the recordset.
I have been attempting to write a macro that grabs financial information from the internet and pastes it into my macro for further analysis. I have been trying to use a query from one of my existing connections (MSN stock quotes).
with my code (below) I have been able make the query pop up but what I can't figure out how to do is to enter anything into the box that pops up. What I'm basically looking for is how to (after the code I have listed below) tell excel to type in certain values into the box that pops up and click "OK" to run the query.
below is my code that initiates the query box asking for tickers
With ActiveSheet.QueryTables.Add(Connection:= _
"FINDER;C:\Program Files\Microsoft Office\Office12\QUERIES\MSN MoneyCentral Investor Stock Quotes.iqy" _
, Destination:=Range("$A$1"))
.Name = "MSN MoneyCentral Investor Stock Quotes"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingAll
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = True
.Refresh BackgroundQuery:=False
End With
Sean, have a looksie at VB.NET as opposed to VBA - you'll find it much more conducive to things like this. As for good libraries for financial data - check out this library:
https://code.google.com/p/yahoo-finance-managed/