Related
I am working on migrating a MS Access Database over to a newer SQL platform.
But, with all of the users who are currently using it, we're migrating slowly/carefully.
The first step is that we are re-writing the VBA code into C#, which is then deployed in a .dll along with the database.
Now, the VBA code calls into the C# to do the business logic, then the VBA continues to do the displays/UI, while Access still hosts the database.
The problem comes in that I have a report that is being run after the business logic from the C# in one place, and apparently MS Access has a cache, which clears every 5 seconds. So, the transaction that occurs in the C# code writes to the database, but the VBA code is still using the cache. This is causing errors, as the records added to the database (which the VBA report is trying to report on) don't exist in the cache yet...
I'm guessing that the C# .dll must be getting treated as a "second connection" to the MS Access database, which is what seems to typically cause this error in my searches (thinks that one process is writing, and the other is reading).
Since the cache is cleared out every 5 seconds, we can just put the process to sleep, and wake it up after 5 seconds, and then run the report, but that's pretty terrible for an end user.
And, making things difficult, the cache seems like it only gets used in the deployed version (so, when running from source / in debug mode, the error never happens).
Doing some searches, there seems to be plenty of people who have said "just refresh the cache." But, the question is: within VBA, how do you refresh the cache?
Any advice would be welcome.
Thanks
I've been fighting the same issue for years as I write a lot of tools around an old Powerbuilder application that has an Access MDB back end.
The cache does exist and it is VERY real. When data is inserted on a different connection than it is queried on, the cache can be directly observed and measured. It was also documented by Microsoft before they blackholed a bunch of their old articles...
Microsoft Jet has a read-cache that is updated every PageTimeout milliseconds (default is 5000ms = 5 seconds). It also has a lazy-write mechanism that operates on a separate thread to main processing and thus writes changes to disk asynchronously. These two mechanisms help boost performance, but in certain situations that require high concurrency, they may create problems.
I've found a couple workarounds that are not the best, but somewhat make due until I find something better or can re-write the app with a better back end database.
The seemingly best answer I've found (that may actually work for you since you say you need VBA) is to use JRO.RefreshCache. I've been trying to figure out how to implement this using C# or VB.net without any luck. Below is a link to a code example where you execute the RefreshCache method on your 2nd connection that needs to pull the data. I have not tested this myself.
https://documentation.help/MSJRO/jrmthrefreshcachex.htm
A workaround I've found that will deliver the query results within 500ms to 1000ms of insert time (instead of anywhere between 500 and 5000 ms - or more):
Use System.Data.ODBC instead of OleDB, with connection string: Driver={Microsoft Access Driver (*.mdb, *.accdb)};Dbq=;
If someone knows how to use the JRO.RefreshCache method with OLEDB and C# or VB.net, I'd be forever grateful. I believe the issue is it's looking for an ADO connection to be passed in, not an OLEDB connection.
I not aware of ANY suggesting that some 5 second cache exits? Where did this idea come from????
Furthermore, if you have 5 users, then you not going to be able to update their cache, are you?
In other words, the issue of some cache for one user still not going to solve or work with mutli-users anyway, is it?
The simple matter is if you load up a form with 100 reocrds, and then other users are ALSO working on that 100 rows, then all users will not see other changes until such time you tell access to re-load the form.
You can do this with a me.Refresh in the form, and then it will show changes made by other users (or even your c# code!!!).
However, that not really the soluion here.
How does near EVERY system deal with this issue?
Answer:
You don't, you "design" the software to take the user work flow into account.
So, in place of loading up a form with 100 rows of data? (which you should not, unless SUPER DUPER reason exists for doing that).
The you provide a UI in which the user FIRST searches for whatever it is they want to work on.
In other words, say you just booked a user on a tour. Now, they call the office back, and want to change some details of that tour. But, a different tour staff might pick up the phone. So, now a 2nd user opens the tour?
So, you solve that issue by NOT loading all the tours into that form in the first place.
you provide a search screen, so they can search for the user, find the user, maybe type in a invoice number or whatever.
You display the results in a pick list, and then launch the form to the ONE record (and perhaps detail records from child tables).
So there no concpet of a cache in Access anymore then there is in c#.
However, if you load up a datatable in c#, and then display that data?
Well, what about the other users on that system. They will not see changes to that data ANY MORE then the current access form.
So, if you want to update some data in c#? Then fine, but you need/want to do two things:
First, before you call any c# code that may update the current form reocrd? You need to FORCE a data save of that current record BEFORE you call any code, be it VBA code, or c# code that going to update that current reocrd the user is working on.
You can in Access save the current reocrd in MANY different ways, but the typical approach is:
' single record save - current record
if me.dirty then me.dirty = false
' VBA or c# code goes here.
' optional refresh the current form to reflect changes
me.Refresh
So, in most cases, it is the "design" of your software that will solve this issue.
For example, in the tour example, or in fact ANY system, the user can't work, can't update, and can't do their job UNLESS they first find/search and have a means to bring up that form + record data in the first place.
So, ANY typical good design will:
Ask the user for that name, invoce number or whatever.
Display the results of the search, and THEN allow the user to pick the record/data to work on. When they are done, they close that form and are RIGHT BACK to the search form to do battle with the next customer or task or phone call or whatever.
So, a search form might look like this:
In above, I typed in smi, and then displayed a pick list.
The user can further type in say part of the first name, and thus now get this:
So, maybe they type in a invoice number, customer number, booking number or whatever.
So, you display the results, and then they can select the row or "thing" to work on.
thus, we click on the row (or above glasses button), and then jump to the ONE record.
so, the user does whatever they have to do with the customer. Now, when done, they close the ONE thing, the ONE main reocrd.
This not only saves the data (so others in the office can now use that booking data), but it also means the data is saved. and they are NOW right back at the search screen, ready to do battle with the next customer.
So, not only does this mean we have a VERY bandwith friednly design (we only pull the one main reocrd into that form), but it also is better for work flow.
The Access form's cache thus becomes a non issue, since we only dealing with the one record.
And as I pointed out, if the system is multi-user, then you NOT going to be able to udpate and deal with multiple users cached data anyway, are you?
Think of ANY system you EVER used from a software point of view.
When you use google, does it download the WHOLE internet, and then you use ctrl-f to search megs and megs of data in the browser?
Nope!
you search first, get a list of that search, and THEN pick one!!
And when that list is display, maybe others on the internet are udpateing, and add new data - but if that was cached in your browser, then it would not work!!!
And same goes for a desktop accounting system. You don't load up all accounts, and THEN have the user go ctrl-f to search all the data. You search for the customer, invoice number and PICK ONE to work on.
And it does not make sense to load up a form with 1000 customers, and then go ctrl-f to find that customer. Same goes for a instant banking machine. It does not download ALL customers and THEN let you search. It asks you FIRST to get what you need. So, be it browser based, desktop based, or JUST ABOUT ANY software you use?
You quite much elminate the cache issue, since not pre-loading boatloads of data, but asking and letting the user search for the data they need.
So, in regards to the Access form data and cache?
If you are on a form, and call VBA code, or c# code or whatever?
If that code update the current form, you have NO MORE OR LESS of a issue when calling VBA code, or c# code!!!! If that code updates the current form, and the reocrd is dirty (has pending edits), then you get that message about the current form's reocrd having been udpated by another user!!!
So, your cache issue does NOT IN ANY WAY exist MORE or LESS as a issue in typical Access software.
As a genreal rule, if you are on a form with pending edits, and say want to pop up some form to edit releated data?
You have to ensure that pending edits are SAVED before you launch an form that can edit the same data, or run code that can/may edit that data.
As a result, ZERO cache issues should exist, and they no more or no less exist when calling sql or VBA update code in a form then calling some c# code from that form.
So, write the pending update for that form.
Then run your VBA, SQL, or c# code.
And then do a me.Refresh to display any changes made by those external routines.
there is no documetjion, or ANY article I can find that suggests some kind of 5 seocnd cache or update - it is a urban myth, and your software challenge here in regards to use c# or VBA, or even SQL server stored procedures?
They are all the same issue, and I dare say that often access is used as a front end to SQL server, and ALL OF the SAME issues exist when using SQL server with ms-access.
I have a system where there are about 30 Excel 2016 templates that all pull prices for the items on each template from a single, master Access 2016 file. These are all located on a server that anyone with access may open. The templates are used to request specific items to be sent to a job site, while also tracking its price so it can be charged to that job. The goal of having a single master file with all the pricing is so that every time a price changes (which is fairly often), I don't have to update the price in every single template.
Right now, I have an Auto_Open module in each of these forms that refreshes the data from the Access file every time a copy of the template is opened. This ensures that the prices for the items that the project managers request for their job is updated every time they use the sheet. This usually completes in less than a second, and it updates all the information just fine. Here's the code:
Sub Auto_Open()
Application.ScreenUpdating = False
Sheets("Import Sheet").Unprotect Password:="secret"
Sheets("Import Sheet").Visible = True 'for some reason the data doesn't like to refresh unless the sheet is visible
ThisWorkbook.RefreshAll
Application.CalculateUntilAsyncQueriesDone 'ensures every connection refreshes
Sheets("Import Sheet").Visible = False
Sheets("Import Sheet").Unprotect Password:="secret"
Application.ScreenUpdating = True
End Sub
Recently, Microsoft put out an update to Excel that combines data imports and transformations into a new single section under the Data tab called "Get and Transform Data." Along with this update, new wizards were released to import data under the "Get Data" button in this section. The old "Legacy Wizards" are still available, which is what I was using and what has been working with the above code.
I looked for information on this new tab, but the only usable info was a Microsoft support web page, and it shows the old organization with separated sections, found here. So, I did my own testing, and I found the following differences:
It's possible to change how the data imports in the new wizard, i.e. I can delete rows and columns that I don't want. (This is particularly valuable for what I'm doing since some of the information for each item isn't used to calculate price, such as the supplier or the individual break down for shipping and the actual cost of the item).
I can't have a password on the new wizard, it doesn't let me open the Access file if it is password-protected. If someone has a solution to this one too, that'd be great.
The connection types are different. In the legacy wizards, the connection type is simply "Microsoft Access Database." However, the new wizards show the connection type as an "OLE DB Query."
Finally, the legacy wizard shows the import as a "connection," while the new wizard imports it as a "query."
My worry is that in the next version of Excel, the legacy wizards will no longer be available, so I would like to update to the new wizards. Further, it would also be beneficial to be able to only import specific rows and columns out of the Access file, which I can't do using the legacy wizards.
However, one of these template sheets imports 11 different tables, which ended up taking about 30 seconds to update when I tried opening a new worksheet based off of the template. This is way too long to be efficient or worth the benefits of the new wizard.
In the connection properties of the tables imported with the new wizard, there's a few options that I messed around with. First, I deleted my Auto_Open module and tried to use the "Refresh data when opening the file" option, this ended up taking a minute to update all of the tables, and it forced me to unlock the sheet, which I would strongly prefer to remain locked. Second, I tried the "Enable Fast Data Load" and "Enable background refresh," and I was able to get it down to 17 seconds, but that still is a pretty long time to wait every time you open a file. I would also prefer not to use background refresh, since the time spent on each sheet may not be enough to guarantee every price is updated, especially on some of the files with 10+ imported tables. Anyway, I attempted ever possible combination and I was only able to reduce the time to, on average, 20 seconds.
I guess the big questions of this post are:
How can I speed up the refresh time for the imported tables when I use the new wizards? And if I can't, how can I ensure that I will still have access to the legacy wizards?
Is it possible to lock an Access file while using the new wizards, and how do I do that?
The CAD technician that I work with is also communicating with some of his contacts within Microsoft to see if he can get any information about the future availability of the legacy wizards. If necessary, I can comment a few example files (with fake prices, obviously) so anyone can mess around with them.
Any help is greatly appreciated. Thank you!
If you can use .xlsm files, then you should use ADODB for connecting to the database.
Here is a good tutorial:
http://www.onlinepclearning.com/import-from-access-to-excel/
You can execute SQL statments to modify the database.
However if your database is password protected you will need to include that in the connection string,
and since it is very easy to crack protected vba projects, the password will not be safe at all.
Currently I am in the process of building an updated Microsoft Access database for our affiliated hospital to keep a timeline of documentation for potential medical studies. The database itself is standalone, so there is no real back end to speak of (which hurts me, but I have no control over how they handle their data). It currently holds only one table with all the necessary fields needed. I'm looking to have reports generate automatically every week, month, quarter, and year. However, I must admit my vba coding abilities are spotty at best. My questions are this:
What code would I need to use to generate said reports?
Would I need to create separate queries just for each type of report based on when it needs to be generated, or can I code it all in one query without the different codes interfering with each other?
If any other information or clarification is needed, I'd be happy to give you what I can.
When calling a Report from VBA or from a macro, two things could be usefull to you:
providing a condition, like
docmd.OpenReport "someReport",acViewPreview,,"[examDate] > Date() - 30"
providing an OpenArg, which can be used by vba in your report to change the periodicity for example ("M" for Monthly and "w" for Weekly"), like
docmd.OpenReport "someReport",acViewPreview,,"[examDate] > Date() - 30",,"W"
Those tricks should allow you to design a report ONCE, then use it with dynamic time range and grouping.
If you don't want to use VBA, for creating automatic reports printing/export you'll need:
Create reports which don't require user entry, use date functions for querying required date intervals.
Create macroses for starting reports and closing the database, no VBA knowledge required
Create tasks in Windows Task Scheduler for starting MS Access at requred days/time. Command line should contain parameters for opening specified database and starting specified macro (/x macro_name). Parameters description you can find, for instance, here
.
The comments say it all.
Access is designed to make report writing of the nature you describe very easy, and a lot can be achieved without using VBA at all or macros for that matter.
If you want to use the same report to display the same sets of data for different date ranges (or perhaps with other changes in criteria), you should prompt the user to enter a date range or the other criteria that will cause the data displayed to be changed. (ie the parameters they entered are used in the query to change the data returned).
This is infinitely preferable to having many many reports and queries that are all the same except they get data for a different set of criteria.
There are many Access Programming books that will help you re-learn these skills. I have found this channel useful https://www.youtube.com/user/ProgrammingMadeEZ/search?query=reports for other Access videos, but it's not as useful for reports.
Harvey
I work with VBA in MS Access databases. I'd like to be able to log when files are saved, modified or deleted without having to update the existing code to do the logging when the pertinent events take place. I want the time, location and the name of the file.
I found a good example here: when file modified
However, it only allows for monitoring a particular location (path). I want to be able to log regardless of where the save, modify or delete takes place. I'm only allowed to program in the MS Office environment in this situation. It seems as though using the Windows API is going to be how this task will be achieved. However, I don't have much experience working with the API. Is there an easier way to achieve what I want that doesn't involve using the API?
Have you worked with After_Updates or After_Insert macros? Also, is your application split? Meaning there's a front-end and a back-end of the database. You can create a separate table that mirrors that table that you need to track changes for. Every time a table is updates, run a macro that inserts a row to that table.
I'm assuming you're saving files to the database. If that's the case, add a after_update or after_insert macro that can keep track of when then files are being modified or added to the table.
I have a series of about 30 Excel reports (.xlsm), which each have a unique connection configured to a database. The connection is just a short SQL script which grabs the data to be shown on the report. The data is populated into a table (not a pivot table).
Every week we need to update these reports, so I use a simple PowerShell script to open each of the files and refresh the connection.
Every so often we need to send the base 30 reports to other work groups so they can manually update the files on their own. This can be a nuisance because some of the reports are very large (30mb+). This makes emailing difficult, and uploading them/downloading them several times a day is just a hassle.
To mitigate this, before we distribute the report templates I try to delete all the rows in the tables, and any unused range. This has helped, but there's still several files that are VERY large (30mb+) even though we've deleted everything in the workbook except the connection, and the empty table.
Through tests, I've realized that if I delete the configured connection, the file size becomes sufficiently small (<1mb) which is what I would expect. This leads me to believe that Excel connections have a sort of cache that needs to be cleared, however I can't find any references for this.
Does anyone know a simple way for reducing the size of a connection in such a way that I could do so programmatically using VBA/Powershell?
If deleting the configured connection reduces your file size enough, you could write a macro to delete your connections and another to reestablish them. As Noldor130884 suggested, you can automatically execute the macros on Workbook_Open and Workbook_Close.
Office Online - Create, Edit & Manage connections to external data
The above reference seems to make the relevant statement below:
"Removing a connection only removes the connection and does not remove any object or data from the workbook."
It looks to me as if the problem is with the formatting. I don't know why but in my files excel reformatted all rows and columns while adding form with data from connection. Thus the sheet was very large but if you check the xml file it shows only formatting data.. Once I deleted manually all "empty" rows the size of the file is normal again. Hope that helps, helped in my case..