Power Query Tables won't download data until VBA finishes executing - vba

I have a spreadsheet with 4 worksheets, each one connected to a power query which connects to an OLEDB database and runs a select to import data into the workbook.
I am trying to automate the refresh of the queries.
I have VBA in the workbook which
- disables background refresh,
- runs Workbook.RefreshAll,
- resets the background refresh to true (as I need this for the automation software to be able to run it in the first place) and
- then waits 30 seconds to give the automation software a bit of time before it starts saving the workbook.
All the software does is call the macro in the workbook and then save following execution.
The queries correctly execute one after the other and I can tell the data in the tables has refreshed.
The power query confirmation that the rows have been downloaded happens after the VBA completes executing, so the automation software saves it before this is done and Excel reports the download as having failed. This isn't a problem when everything works, but we won't know when the download has truly failed, unless I add in some sort of a row count comparison which is really a workaround rather than an actual solution.
I have tried creating two subs in the workbook, one to do the refresh and then another which calls the first sub and then waits 30 seconds, I was hoping the download would complete after the first sub finishes executing, but this doesn't happen. I have tried refreshing one connection at a time instead of using RefreshAll, tried refreshing the actual power query tables instead of the connections and have tried refreshing with background query set to true and then looping until the refresh is marked as complete (which works if you step through it, but crashes if you just run the VBA in full).
I also tried saving the spreadsheet at the end of the VBA, but it still waits until the save is complete to update the power query status.
My latest VBA:
Sub Workbook_RefreshAll()
Application.DisplayAlerts = False
For Each objConnection In ThisWorkbook.Connections
'Temporarily disable background-refresh
objConnection.OLEDBConnection.BackgroundQuery = False
objConnection.OLEDBConnection.MaintainConnection = False
Next
ActiveWorkbook.RefreshAll
For Each objConnection In ThisWorkbook.Connections
'Re-enable background-refresh
objConnection.OLEDBConnection.BackgroundQuery = True
objConnection.OLEDBConnection.MaintainConnection = True
Next
newHour = Hour(Now())
newMinute = Minute(Now())
newSecond = Second(Now()) + 30
waitTime = TimeSerial(newHour, newMinute, newSecond)
Application.Wait waitTime
End Sub
(I can't remember whether the maintain connection parameter actually helped as I as trying to force the connection to drop after each refresh, but it works so haven't taken it out.)

Instead of Application.Wait try the following:
Do until now() >= waittime
DoEvents
Loop

Related

Automatic refreshment of Excel DDE RTD

I'm subscribed to a RTD financial application with the option to export real time data to Excel via DDE. So, using the DDE copy/paste app option, I've created a Excel table with real time DDE financial data. Cells actually contains DDE formulas. Now I open, refresh and save table manually, but what I need is a solution that will open, refresh the data, save and close this excel table automatically.
I didn't now how to attach the file, so here is a screenshot of it.
My Excel DDE file
Since I'm new to vba and vbscript, I was searching internet for a solution but without any success, especially not for automatic update (refreshment) of the DDE data.
What I've been doing for the past few days, was searching, trying and customising different kind of vba macros and vbscripts, and today I'm completely lost and confused.
I have tried bunch of suggested solution, even yours
Refresh data and exit with saving Macro Excel
and On workbook open, Excel Macro to refresh all data connections sheets and pivot tables and then export the pivot to csv but in my case they don't work.
Any solution, reference, code example will be greatly appreciated.
Kindly try the below one and
be careful when using
Application.Calculation = xlCalculationManual
Application.Calculation = xlCalculationAutomatic
this sets the entire excel application to calculate formula's either automatically or manually. If you use
Application.Calculation = xlCalculationManual
you'll notice your automatic formulas no longer work.
Thanks for the reply #mulla, but this doesn't work. In vbscript I get runtime error and in vba nothing happens (does not affect refreshment). The same thing is for the Worksheet.Calculate.
Than I recorded a macro upon manual update of the links (data) in order to inspect how Excel proceed update and this actually works, but not all the time. If I open Workbook manually and I keep it opened than whole export works perfectly since the data is constantly refreshed (not what I need). In case when the Workbook is opened via vbscript only (what I need), as you can see in the output below some of the values are exported successfully (for the faster refreshment I believe) and some are #N/A (for the slower refreshment). All the data is refreshed every second, except upon opening the file where successfully exported values are refreshed instantly and #N/A values need few seconds (2-3s) to be refreshed.
Recorder macro uses ActiveWorkbook.UpdateLink as follows (the code below is for the first row, but the same logic is used and for the rest of the rows):
Sub PriceUpdate()
ActiveWorkbook.UpdateLinks = xlUpdateLinksAlways
ActiveWorkbook.UpdateLink Name:= _
"vegadde|VEGA!897789,148,1#""1,\""12,0,0\"""",1", Type:=xlOLELinks
ActiveWorkbook.UpdateLink Name:= _
"vegadde|VEGA!897789,148,1#""1,\""12,0,0\"""",3", Type:=xlOLELinks
ActiveWorkbook.UpdateLink Name:= _
"vegadde|VEGA!897789,148,1#""1,\""12,0,0\"""",4", Type:=xlOLELinks
...
End Sub
Then I use vbscript to call the macro and export data in CSV comma delimited file with timestamp name:
Option Explicit
Dim objExcel, objBook, objSheet
Set objExcel = CreateObject("Excel.Application")
Set objBook = objExcel.Workbooks.Open ("d:\exptest\exptest.xlsm", 0, False)
'Set objSheet = objBook.Worksheets.Item(1)
objExcel.DisplayAlerts = False
objExcel.Run "PriceUpdate"
WScript.Sleep 5000 ' Delay in order to update links
objBook.SaveAs "d:\exptest\" & Year(Now) & "." & Month(Now) & "." & Day(Now) & "_" & Hour(Now) & "-" & Minute(Now) & ".txt",6
objBook.Close False
objExcel.DisplayAlerts = True
objExcel.Quit
'Set objSheet = Nothing
Set objBook = Nothing
Set objExcel = Nothing
And what I get is the following uotput:
Inst,Price,Datetime
USD,1.1015,7/22/2016 12:48
GBP,#N/A,#N/A
CHF,#N/A,#N/A
SEK,9.4962,7/22/2016 12:48
NOK,#N/A,#N/A
JPY,#N/A,#N/A
DKK,#N/A,#N/A
CAD,#N/A,#N/A
AUD,1.47395,7/22/2016 12:48
RUB,71.0082,7/22/2016 12:48
I'm struggling to learn Excel logic behind different updates in order to solve my problem and to get desired solution, but with no success.
And this is only a test file with 22 links. The real one has 482 links that need to be updated.
Marjan. Not sure if this is still a problem for you, but I ran into a similar problem with refreshing RTD data in a closed Excel workbook and seem to have found a solution. As you mentioned, when you open the workbook, the data refreshes automatically (with some of the more complex data points taking 2-3 sec). The problem is, you don't want to have to constantly manually open a workbook to refresh data. And I've found that the data fails to refresh with methods like RTD.RefreshData called to a closed workbook.
So to fix this, I created a vba module that simply opens the workbook, refreshes it, pauses for a second (you may need 2-3 more seconds for all data to update) and then saves and closes the workbook. This method is called recursively. Essentially this does the work of constantly opening and closing the workbook, while delaying briefly so that the RTD data can refresh. So far, this is able to provide refreshed RTD data successfully (although there is an inherent slight delay of a few seconds). Here's the basis of my code:
Sub refreshXLS()
Path = "workbook.xlsm" 'the workbook path you want to refresh
Workbooks.Open Path
ActiveWorkbook.RefreshAll
Application.OnTime Now() + TimeValue("00:00:01"), "closeActive"
End Sub
Sub closeActive()
ActiveWorkbook.Save
ActiveWorkbook.Close
refreshXLS
End Sub
Hope this helps if you still wish to solve this problem!

Run macro automatically via vbscript from timer

I have a macro that downloads some information from the internet and I need to schedule it to run automatically starting at a certain time of day and running repeatedly both at set time intervals although occasionally at specific times output by the macro.
I have approached the problem in two ways, both of which have presented problems that I can't figure out. The first approach is using a timer sub that runs the macro using the Application.OnTime command with the runtime derived from a cell in the workbook that is updated by the macro.
This method works pretty well if I am on the computer doing something else. However, I have noticed that if I leave, even if the computer doesn't go to sleep, the program inevitably stops re-running after 30min or so. Sometimes it also stops re-running even when I'm active on the computer. Why is this? What could be causing this? Also, naturally, I can't schedule the program to open and run automatically from within the macro, so I turned to the VBScript & Task Scheduler method for this part of the solution.
I have, however, encountered two problems with the execution of the VBScript. First, is that it often fails to start when run by Task Scheduler. Second, is that when it does run, the Application.OnTime command within the macro doesn't work. This is a problem because the macro needs to run sometimes at irregular time intervals which are determined during the course of the day as the macro runs through its iterations. Why would the vbscript fail to run when task manager tries, and why would it not initiate the application.ontime command the way the macro does when I run it manually?
VBA Timer Code:
Sub Timer()
Dim Runtime As Date
If Time <= TimeValue("17:45:00") Then
Workbooks("10am_Swing_Report").Sheets("Parameters").Cells(17, 2).Value = 10
ElseIf Time > TimeValue("17:45:00") And Time <= TimeValue("18:15:00") Then
Workbooks("10am_Swing_Report").Sheets("Parameters").Cells(17, 2).Value = 13
ElseIf Time > TimeValue("18:15:00") And Time <= TimeValue("18:45:00") Then
Workbooks("10am_Swing_Report").Sheets("Parameters").Cells(17, 2).Value = 16
ElseIf Time > TimeValue("18:45:00") And Time <= TimeValue("19:15:00") Then
Workbooks("10am_Swing_Report").Sheets("Parameters").Cells(17, 2).Value = 19
End If
Workbooks("10am_Swing_Report").Sheets("Parameters").Range("B16").Calculate
If Time > Workbooks("10am_Swing_Report").Sheets("Parameters").Range("B16").Value Then
Runtime = Time + TimeValue("00:00:30")
Else
Runtime = Workbooks("10am_Swing_Report").Sheets("Parameters").Range("B16").Value
End If
Application.OnTime TimeValue(Runtime), "Swingtimer"
Workbooks("10am_Swing_Report").Sheets("Parameters").Range("C16").Value = Runtime
Workbooks("10am_Swing_Report").Save
End Sub
Thanks!
I've updated the Vbs code to include a validation that the runtime has been passed, but I'm still encountering a problem with Task Scheduler failing to start the task. I'm getting the following error: "Launch request ignored, instance already running" followed by: "Task Start Failed". I was originally thinking this was because the Vbscript was still running from the previous instance, but I'm getting this from the very beginning.
Option Explicit
Dim xlApp
Dim xlBook
Dim runtime
Dim xlSheet
Set xlApp = CreateObject("Excel.Application")
xlApp.DisplayAlerts = False
'xlApp.Application.DisplayAlerts = True
Set xlBook = xlApp.Workbooks.Open("C:\Users\DORIAN\Dropbox\Swing_Report.xlsm",1,False)
Set xlSheet = xlBook.Sheets("Parameters")
'xlBook.application.Visible = True
'MsgBox FormatDateTime(xlSheet.Range("B16").Value,3)
runtime = TimeValue(FormatDateTime(xlSheet.Range("B16").Value,3))
'MsgBox runtime
If Time > runtime Then
xlApp.Run "Swingtimer"
'MsgBox "Running..."
WScript.Sleep 30000
Else
xlApp.Application.OnTime runtime, "Swingtimer"
'MsgBox "Timer Set"
End If
xlbook.close True
WScript.Quit
Could you do it in a different way?
1)
Run vbscript it in Windows Scheduler say every 1 minute or 1 hour or ….
2)
Put some validation inside VBscript
If Condition then
Execute your code
Else
Stop vbscript
End if
Using this you probably can get rid of OnTime event handler
Best regards
Alex
I think you are over complicating things. You just need a Workbook_Open event and Windows Task Scheduler.
Something like this (customize to suit your needs).
Private Sub Workbook_Open()
Msgbox Date
Worksheets("Sheet1").Range("A1").Value = Date
End Sub
Schedule the event using the Windows Task Scheduler.
http://www.sevenforums.com/tutorials/11949-elevated-program-shortcut-without-uac-prompt-create.html

VBA: To calculate the worksheet in background, while message userform is active

I wrote a loop procedure in vba, which pastes a lot of data each time to the new worksheet. Actually it pastes the formulas that get calculated. It takes excel approximately 40 seconds to get all calculations done. Unfortunately the next loop does not wait 40 seconds and fills with the data the next sheet. It takes a lot of resources. To make a loop to wait a little bit, but still to perform calculations I used userform, which appears for 40 seconds, than it will be unloaded and is used in the next loop. The purpose of the userform is to allow the calculations to get done before the calculations in the next sheet begin. This strategy has a drawback, since when the useform is active, the calculations in the background freeze. One can change the options of the useform to showmodal false, but then I cannot upload and unload it each time as new loop begins. Besides I am not sure weather the formulas a still calculated in the inactive sheet. I have to mention that the formulas in the sheet each time activate an excel add-in and I cannot make an add in to wait with calculations, even if the option manual calculations is turned on.
If anybody knows how to postpone the loop but not to make complete excel application wait (Application.wait or ontime methods do not work) I will appreciate the help a lot. I tried also to active and deactivate events, switch to manual calculation and back. Also this does not help and all data a calculated at once.
The method described in the other post did not worked for me and all formulas are still calculated at the same time:
Application.Calculate
If Not Application.CalculationState = xlDone Then
DoEvents
End If
My original code that copies formulas to the new worksheet:
For raw_i = 1 To 3
Set o = GetObject("d:\formulas.xlsx")
Set Sheet_i = Worksheets.Add(after:=Worksheets(Worksheets.Count))
Sheet_i.Cells(1, 1).Formula = o.Worksheets("Sheet1").Cells(raw_i, 1).Formula
Set o = Nothing
LoadForm
next raw_i
VBA doesn't support multiple process threads - there is one execution route and if you pause it, then you pause the whole execution. It's all or nothing.
Best hope you've got is a While Loop:
While Application.CalculationState <> xlDone
DoEvents
Wend
Which will loop until calculations have finished.
In response to your most recent edit:
Try changing the main loop to:
For raw_i = 1 To 3
Set o = GetObject("d:\formulas.xlsx")
Set Sheet_i = Worksheets.Add(after:=Worksheets(Worksheets.Count))
Sheet_i.Cells(1, 1).Formula = o.Worksheets("Sheet1").Cells(raw_i, 1).Formula
Sheet_i.Calculate
DoEvents
Set o = Nothing
Next

VBA While Loop freezing data uplink, need to run Sub once a second

I'm needing to run a sub once a second while also receiving data from a feed. The function looks at the data in the cells, determines if conditions are right, and if so sends a command to the trade server (where my data feed is coming in). This issue is once I run the loop, the While Loop freezes my cells in excel, making the program execute without updating.
Dim timeremaining As Integer
timeremaining = Range("E7").Value
Application.Calculation = xlCalculationAutomatic
Do While timeremaining > 5
Endowment
Sleep 1000
Loop
I've tried the setting a timer (OnTime) approach, and creating a function to run it a set number of times, yet both functions freeze my data feed.
Thank you.
Try using the wait method in VBA
For example:
Application.Wait(Now + TimeValue("0:00:10"))

VBA: Handling PowerPivot refreshing error

I'm using a Macro to refresh some data in PowerPivot, it then autosaves the updated file on my computer.
My problem is that I'm using SSH in order to connect to a database and due to bad connection sometimes it crashes and stop the refreshing process. Then powerpivot can't refresh the worksheet but my macro keep being executed as if nothing happened...
I'd like to know if there is a way to handle in VBA such an error in order to loop the refreshing process until it is fully and properly executed.
The simplest and most straight forward way I would suggest is to use a boolean variable and assign in to yes if the powerPivot gets refreshed properly and false if it doesn't. Thus only when the variable is true the macro will run. An example is:
Dim result As Boolean
result = False
if PowerPivot gets refreshed then
'the refesh code
result = true
end if
if result = true then
'run the macro code
end if
Thus only when powerpivot is fully refreshed the macro will run. Otherwise not.