Running SAS Stored Process in Excel fails after two runs - vba

I hope this is an appropriate place to ask this question.
I have recently built a data analysis tool in Excel that works by submitting inputs to a SAS Stored Process (as an 'input stream'), running the processes and displaying the results in Excel.
I also use some code to check for and remove all active stored processes from the workbook before running the process again.
This runs successfuly the first 2 times, but fails on the third attempt. It always fails on the third attempt and I can't figure out why.
Is there some kind of memory allocation for Excel VBA that's exhausted by this stage? Or some other buffer that's maxed out? I've stepped-in to every line of the VBA code and it appears to hang (on the third run) at the following line:
SAS.InsertStoredProcess processLoc, _
outputSheet.Range("A1"), , , inputStream
Code used to initiate SAS Add-in for Microsoft Office:
Dim SAS As SASExcelAddIn
Set SAS = Application.COMAddIns.Item("SAS.ExcelAddIn").Object
Code used to delete stored processes from target output sheet:
Dim Processes As SASStoredProcesses
Set Processes = SAS.GetStoredProcesses(outputSheet)
Dim i As Integer
For i = 1 To Processes.Count
' MsgBox Processes.Item(i).DisplayName
Processes.Item(i).Delete
Next i
Code used to insert and run stored process:
Dim inputStream As SASRanges
Set inputStream = New SASRanges
inputStream.Add "Prompts", inputSheet.Range("DrillDown_Input")
SAS.InsertStoredProcess processLoc, _
outputSheet.Range("A1"), , , inputStream
Cheers

On reflection, my theory here would be that you are hitting the limit of multibridge connections. Each multibridge connection represents a port, and the more ports you have the more parallel connections are enabled. By default there are three, perhaps you have two, or you are kicking off another STP at the same time?
This would explain the behaviour. I had a spreadsheet that called STPs and it would always fail on the fourth call because the first three were running. You can get around this by either a) increasing the number of multibridge connections or b) chaining your processes so they run sequentially.

I don't know if this'll be useful, but I had a similar problem running a DLL written in C++ through VBA. The problem occurred because the function in the DLL returned a double value, which I didn't need and so the code in VBA did
Call SomeProcessFromDLL()
But the process was returning a double floating point value which was 'filling up' some buffer memory in VBA and VBA has a limited buffer (I think it gave up at 8 tries). So the solution for me was
Dim TempToDiscard as Double
TempToDiscard = SomeProcessFromDLL()
Maybe looking at the documentation of the process being called would help here, especially if it's returning some value to be discarded anyway, like a
Return 0;

I never liked using the IOM in VBA, mainly due to issues with references and having to do client installs when rolling out applications. Last year I found a MUCH better way to connect Excel and SAS - using the Stored Process web application. Simply set up your server side SAS process with streaming output, and pass your inputs via an Excel Web query. No client installs, no worries about SAS version upgrades, hardly any code - am surprised it's not used more often!
See: http://rawsas.blogspot.co.uk/2016/11/sas-as-service-easy-way-to-get-sas-into.html

Related

LOF function in Visual Basic suddenly not working

I am doing some analysis on the US data customer credit card complaints database - see https://catalog.data.gov/dataset/consumer-complaint-database
Saving the file to a csv and running the following code leads to an error
Sub test()
Dim FolderPath as String
Dim strfilename as String
Dim strtextline as String
strfilename = "C:\Work\VB\complaints.csv"
Open strfilename For Input As #1
strFileContent = Input(LOF(iFile),1)
Close #1
End Sub
Runtime error '5'
Invalid Procedure Call or Argument
Is this an error with Visual Basic on my installation? I'm running VBA on Excel with Microsoft Office Professional Plus 2019 with Windows 10.
The file is simply too large to read it in one chunk - its size is 1.5 GB and it holds more than 3 million rows.
You could rewrite your routine to read the file in chunks or row by row, but before you do so, you need to make up your mind what you want to do with the data. You cannot store them in Excel - it's maximum row number is a little bit above 1m (1'048'576 to be precise) and even with this limitation, it's nearly impossible to work. If Excel doesn't crash at all, it will be painfully slow.
My advice would be to store the data in a real database. There are a lot of free db systems around, eg SQL server, PostGres, MariaDB... You need of course learn about databases and SQL to handle that, but I don't see any alternative - Excel, as said, is not build and not capable to handle that amount of data.

Access not exiting

Recently, my Access .mdb database started intermittently not allowing Access (both Access 2003 and 2007) to quit. If I quit (whether by pressing the X button or from the menu, then it closes the database and appears to exit Access, as well, but then it suddenly reappears (without any database open). The only way for me to exit at that point is from the task manager.
There are two significant changes that I did recently that might be related. 1) I started using the WinSCP .Net assembly to access an ftp server, which I had to install and register for COM from the instructions here. 2) I started using ODBC, first as a linked table, and then from VBA ADO code (see this). I doubt that this second change is causing this problem because I've had the problem both when I was using the linked tables and when with ADO.
This doesn't happen every time I open the database, and I haven't noticed a pattern. What could be causing this strange problem?
Edit - I found the root of the problem. By breaking my ftp download code at various points and seeing whether it will exit, I narrowed it down to the following:
Dim PDFFolders As Recordset
Set PDFFolders = CurrentDb.OpenRecordset("PDFFolders")
Dim syncOptions As TransferOptions
Set syncOptions = New TransferOptions
syncOptions.filemask = "*/*.pdf"
On Error Resume Next 'In case it doesn't exist
Do While Not PDFFolders.EOF
sess.SynchronizeDirectories SynchronizationMode_Local, info!RTFFolder, _
info!BasePDFFolder & "/" & PDFFolders!Name, False, , , _
syncOptions
PDFFolders.MoveNext
Loop
PDFFolders.Close
Set syncOptions = Nothing
Set PDFFolders = Nothing
On Error GoTo 0
If it runs the sess.SynchronizeDirectories statement, then access won't exit, otherwise it does. Looks to me like a bug win WinSCP.
I can do other things like downloading files, creating directories, etc. with no problem, but when it gets to this statement, it doesn't exit access afterwards.
Sticky instances of Access usually result from hanging object references.
If Access hangs the way you described, I would suspect a nasty circular reference.
To investigate on circular references, you basically have two options:
Inspect your code on circular dependencies - That might become tedious and not so easy. But if you know your code base deeply, you might have suspects where to look first.
Add logging to your code - In case you cannot spot the problem via inspection alone, you can consider adding consequent logging of object creation/deletion (through Class_Initialize/Class_Terminate). For larger code bases using classes heavily, this is a good investment to start with.
If your problem is with classes where you cannot control the code (as is your case), you might consider using that external classes only through wrapper classes where you can log creation/deletion. Of course in tricky cases, termination of the wrapper class does not mean termination of the wrapped class.
BTW, I strongly recommend to make sure to set every object reference explicitly to Nothing ASAP:
Set MyObj = GetMyObject()
' Proceed with coding here later
' First write the following line
Set MyObj = Nothing
Special thoughts have to be given in the case of local error handling to make sure to set the reference to Nothing in either case.
A good way to ensure this is using a With-block instead of an explicit variable (if the usage pattern allows to):
With GetMyObject()
' Use the object's members here
End With
With this pattern you save declaring the local variable and can be sure that the object reference does not survive the current method.
I still think it's a bug in WinSCP, but I found a workaround. I noticed that it only happened if I took the information from a table in the database, and not if I put in a hard-coded string. So I just added & vbNullString, which concatenates a blank string, which changes the data type from a Field to a String, and now it doesn't happen anymore.

VBA Word 2010 Paste Error / No Wait Command

I have a project that takes several documents as inputs, does some processing on them, and creates several new documents at the end. I am currently running into problems with pasting content from one Word document into another. The following code snippet seemed relevant:
Set refOrigin = FindReference(OriginDoc)
Set refDest = PasteDoc.Range(PasteDoc.Content.Start, PasteDoc.Content.End)
refDest.Collapse wdCollapseEnd
refOrigin.Copy
refDest.Paste
When running this code, I will occasionally get Run-time error 4198, Command Failed at the paste line in the code. However, when I go into the debugger, I can see that both refDest and refOrigin are valid ranges. Furthermore, when I step through the code line-by-line, it works. However, I can tell that in the instance where it failed, it inserted an embedded Word document already.
I've done some research on the issues and I believe that there is some type of problem of the code running to fast for the clipboard to keep up with it sometimes. This makes sense to me because when I run the macro from a document on a network drive, it runs without a hitch.
I thought that I would be able to simply add a wait command with Application.Wait, but it turns out that Word 2010 doesn't support this command; it's only in Excel.
Does anyone have ideas as to the root of this problem, possible solutions, or any way to give Word 2010 a wait command? Thanks.
For completeness, the following code mimics the Excel `Application.Wait' method from this question.
Dim tmpStart
tmpStart = Timer
Do
DoEvents
Loop While (tmpStart + 1) > Timer

"System resource exceeded" when updating recordset, and possibly at other points

I'm experiencing an intermittent issue with an application that's Excel 2010 front-end, Access 2010 back end. It's in use by 5-10 users simultaneously. Recently, users have started intermittently receiving the following error:
Run-time error '3035': System resource exceeded.
Sometimes the Debug button is grayed out so I can't jump to the code that caused the error, but when it's available to click, it takes me to the following code:
'Open connection to back end DB
Set db = OpenDatabase(dbPath)
'Open a recordset of a table
Set RS = db.OpenRecordset(Tbl)
'loop through rows in a 2D array
For i = FR To LR
RS.AddNew
'loop through columns of the 2D array
For j = 1 to LC
'set values for various fields in the new record, using values from the array
Next
RS.Update
Next
Here, the RS.Update is marked as the line that's causing the error.
What's odd is that this problem comes and goes; users will repeatedly receive it when attempting to submit a certain data set, then, several hours later, when they try to submit the same data set again, the operation succeeds without the error. It's also perplexing that sometimes the Debug button is available and sometimes it isn't.
One issue might be the size of the Access back end; it's currently ~650 MB, and we didn't start getting these messages until it grew to around 600 MB.
Any ideas as to what could be causing this? Various Google hits indicate that this problem sometimes happens when a join query has too many fields, but this is just a recordset of a table, not a join query.
Ahh, methink that this is one of the strange errors that pop-up when you write a lot to a backend database that can't keep up with the management of the lock file.
The solution is to make sure you keep a connection open to the back-end database from each of your client and that you hold onto that connection until you close the client.
Just open a recordset to a table (say a dummy table with only one record), and keep that recordset open until you close the application.
Resource-wise, keeping this connection open will have no detrimental effect on performance or memory consumption, but it will ensure that the lock file is not continuously created/deleted every time a connection is open then closed.
Keeping that connection open will also substantially increase the performance of your data access.
Edit:
You should be more explicit when using recordsets and specify exactly the mode of operation you need:
Set RS = db.OpenRecordset(Tbl, dbOpenTable, dbFailOnError)
or, faster if you are only appending data:
Set RS = db.OpenRecordset(Tbl, dbOpenTable, dbAppendOnly + dbFailOnError)
Also make absolutely sure you close the recordset once you're finished with appending the data!:
Set RS = db.OpenRecordset(Tbl, dbOpenTable, dbAppendOnly + dbFailOnError)
With RS
'loop through rows in a 2D array
For i = FR To LR
.AddNew
'loop through columns of the 2D array
For j = 1 To LC
'set values for various fields in the new record,
'using values from the array
Next
.Update
Next
.Close
End With
Set RS = Nothing
This is caused by running out of available virtual memory (VM) aka swap disk. A 32 bit app cannot use more than 2gb and for some reason Access uses a lot of VM and when it needs more and cannot get any then you run out of system resources.
Solution is to make sure your VM is at least 4 times the RAM and to restart your PC at least daily, only this clears out the VM from garbage left lying around from other apps.
You will never have had this issue on a 32 bit OS, its only now with 64 bit OS that this happens.
Short Story
My 3035 error was caused by an add.new to a file without a primary key.
Composing a copy of the file with a primary key assigned and replacing the key-less file appears to have corrected the problem.
Long Story
I have been experiencing 3035 error on an Add.New to an existing backend table with >81,000 records. After searching the web for ideas and coming up dry I reflected on possible issues.
I compacted/repaired the backend files to no affect. Then decided to check the file design. It turns out there was no primary key assigned.
Assigning a primary key to the auto number ID field caused the same 3035 error! So I copied the data structure to a new file, assigned a primary key to the new file and then did a query append of the original file to the new file. Finally I renamed the files.
Using the new file appears to be working.

Intermittent error when attempting to control another database

I have the following code:
Dim obj As New Access.Application
obj.OpenCurrentDatabase (CurrentProject.Path & "\Working.mdb")
obj.Run "Routine"
obj.CloseCurrentDatabase
Set obj = Nothing
The problem I'm experimenting is a pop-up that tells me Access can't set the focus on the other database. As you can see from the code, I want to run a Subroutine in another mdb. Any other way to achieve this will be appreciated.
I'm working with MS Access 2003.
This is an intermittent error. As this is production code that will be run only once a month, it's extremely difficult to reproduce, and I can't give you the exact text and number at this time. It is the second month this happened.
I suspect this may occur when someone is working with this or the other database.
The dataflow is to update all 'projects' once a month in one database and then make this information available in the other database.
Maybe, it's because of the first line in the 'Routines' code:
If vbNo = MsgBox("Do you want to update?", vbYesNo, "Update") Then
Exit Function
End If
I'll make another subroutine without the MsgBox.
I've been able to reproduce this behaviour. It happens when the focus has to shift to the called database, but the user sets the focus ([ALT]+[TAB]) on the first database. The 'solution' was to educate the user.
This is an intermittent error. As this is production code that will be run only once a month, it's extremely difficult to reproduce, and I can't give you the exact text and number at this time. It is the second month this happened.
I suspect this may occur when someone is working with this or the other database.
The dataflow is to update all 'projects' once a month in one database and then make this information available in the other database.
Maybe, it's because of the first line in the 'Routines' code:
If vbNo = MsgBox("Do you want to update?", vbYesNo, "Update") Then
Exit Function
End If
I'll make another subroutine without the MsgBox.
I've tried this in our development database and it works. This doesn't mean anything as the other code also workes fine in development.
I guess this error message is linked to the state of one of your databases. You are using here Jet connections and Access objects, and you might not be able, for multiple reasons (multi-user environment, unability to delete LDB Lock file, etc), to properly close your active database and open another one. So, according to me, the solution is to forget the Jet engine and to use another connexion to update the data in the "other" database.
When you say "The dataflow is to update all 'projects' once a month in one database and then make this information available in the other database", I assume that the role of your "Routine" is to update some data, either via SQL instructions or equivalent recordset updates.
Why don't you try to make the corresponding updates by opening a connexion to your other database and (1) send the corresponding SQL instructions or (2) opening recordset and making requested updates?
One idea would be for example:
Dim cn as ADODB.connexion,
qr as string,
rs as ADODB.recordset
'qr can be "Update Table_Blablabla Set ... Where ...
'rs can be "SELECT * From Table_Blablabla INNER JOIN Table_Blobloblo
set cn = New ADODB.connexion
cn.open
You can here send any SQL instruction (with command object and execute method)
or open and update any recordset linked to your other database, then
cn.close
This can also be done via an ODBC connexion (and DAO.recordsets), so you can choose your favorite objects.
If you would like another means of running the function, try the following:
Dim obj As New Access.Application
obj.OpenCurrentDatabase (CurrentProject.Path & "\Working.mdb")
obj.DoCmd.RunMacro "MyMacro"
obj.CloseCurrentDatabase
Set obj = Nothing
Where 'MyMacro' has an action of 'RunCode' with the Function name you would prefer to execute in Working.mdb
I've been able to reproduce the error in 'development'.
"This action cannot be completed because the other application is busy. Choose 'Switch To' to activate ...."
I really can't see the rest of the message, as it is blinking very fast. I guess this error is due to 'switching' between the two databases. I hope that, by educating the user, this will stop.
Philippe, your answer is, of course, correct. I'd have chosen that path if I hadn't developed the 'routine' beforehand.
"I've been able to reproduce this behaviour. It happens when the focus has to shift to the called database, but the user sets the focus ([ALT]+[TAB]) on the first database. The 'solution' was to educate the user." As it is impossible to prevent the user to switch application in Windows, I'd like to close the subject.