I have a series of about 30 Excel reports (.xlsm), which each have a unique connection configured to a database. The connection is just a short SQL script which grabs the data to be shown on the report. The data is populated into a table (not a pivot table).
Every week we need to update these reports, so I use a simple PowerShell script to open each of the files and refresh the connection.
Every so often we need to send the base 30 reports to other work groups so they can manually update the files on their own. This can be a nuisance because some of the reports are very large (30mb+). This makes emailing difficult, and uploading them/downloading them several times a day is just a hassle.
To mitigate this, before we distribute the report templates I try to delete all the rows in the tables, and any unused range. This has helped, but there's still several files that are VERY large (30mb+) even though we've deleted everything in the workbook except the connection, and the empty table.
Through tests, I've realized that if I delete the configured connection, the file size becomes sufficiently small (<1mb) which is what I would expect. This leads me to believe that Excel connections have a sort of cache that needs to be cleared, however I can't find any references for this.
Does anyone know a simple way for reducing the size of a connection in such a way that I could do so programmatically using VBA/Powershell?
If deleting the configured connection reduces your file size enough, you could write a macro to delete your connections and another to reestablish them. As Noldor130884 suggested, you can automatically execute the macros on Workbook_Open and Workbook_Close.
Office Online - Create, Edit & Manage connections to external data
The above reference seems to make the relevant statement below:
"Removing a connection only removes the connection and does not remove any object or data from the workbook."
It looks to me as if the problem is with the formatting. I don't know why but in my files excel reformatted all rows and columns while adding form with data from connection. Thus the sheet was very large but if you check the xml file it shows only formatting data.. Once I deleted manually all "empty" rows the size of the file is normal again. Hope that helps, helped in my case..
Related
I have a system where there are about 30 Excel 2016 templates that all pull prices for the items on each template from a single, master Access 2016 file. These are all located on a server that anyone with access may open. The templates are used to request specific items to be sent to a job site, while also tracking its price so it can be charged to that job. The goal of having a single master file with all the pricing is so that every time a price changes (which is fairly often), I don't have to update the price in every single template.
Right now, I have an Auto_Open module in each of these forms that refreshes the data from the Access file every time a copy of the template is opened. This ensures that the prices for the items that the project managers request for their job is updated every time they use the sheet. This usually completes in less than a second, and it updates all the information just fine. Here's the code:
Sub Auto_Open()
Application.ScreenUpdating = False
Sheets("Import Sheet").Unprotect Password:="secret"
Sheets("Import Sheet").Visible = True 'for some reason the data doesn't like to refresh unless the sheet is visible
ThisWorkbook.RefreshAll
Application.CalculateUntilAsyncQueriesDone 'ensures every connection refreshes
Sheets("Import Sheet").Visible = False
Sheets("Import Sheet").Unprotect Password:="secret"
Application.ScreenUpdating = True
End Sub
Recently, Microsoft put out an update to Excel that combines data imports and transformations into a new single section under the Data tab called "Get and Transform Data." Along with this update, new wizards were released to import data under the "Get Data" button in this section. The old "Legacy Wizards" are still available, which is what I was using and what has been working with the above code.
I looked for information on this new tab, but the only usable info was a Microsoft support web page, and it shows the old organization with separated sections, found here. So, I did my own testing, and I found the following differences:
It's possible to change how the data imports in the new wizard, i.e. I can delete rows and columns that I don't want. (This is particularly valuable for what I'm doing since some of the information for each item isn't used to calculate price, such as the supplier or the individual break down for shipping and the actual cost of the item).
I can't have a password on the new wizard, it doesn't let me open the Access file if it is password-protected. If someone has a solution to this one too, that'd be great.
The connection types are different. In the legacy wizards, the connection type is simply "Microsoft Access Database." However, the new wizards show the connection type as an "OLE DB Query."
Finally, the legacy wizard shows the import as a "connection," while the new wizard imports it as a "query."
My worry is that in the next version of Excel, the legacy wizards will no longer be available, so I would like to update to the new wizards. Further, it would also be beneficial to be able to only import specific rows and columns out of the Access file, which I can't do using the legacy wizards.
However, one of these template sheets imports 11 different tables, which ended up taking about 30 seconds to update when I tried opening a new worksheet based off of the template. This is way too long to be efficient or worth the benefits of the new wizard.
In the connection properties of the tables imported with the new wizard, there's a few options that I messed around with. First, I deleted my Auto_Open module and tried to use the "Refresh data when opening the file" option, this ended up taking a minute to update all of the tables, and it forced me to unlock the sheet, which I would strongly prefer to remain locked. Second, I tried the "Enable Fast Data Load" and "Enable background refresh," and I was able to get it down to 17 seconds, but that still is a pretty long time to wait every time you open a file. I would also prefer not to use background refresh, since the time spent on each sheet may not be enough to guarantee every price is updated, especially on some of the files with 10+ imported tables. Anyway, I attempted ever possible combination and I was only able to reduce the time to, on average, 20 seconds.
I guess the big questions of this post are:
How can I speed up the refresh time for the imported tables when I use the new wizards? And if I can't, how can I ensure that I will still have access to the legacy wizards?
Is it possible to lock an Access file while using the new wizards, and how do I do that?
The CAD technician that I work with is also communicating with some of his contacts within Microsoft to see if he can get any information about the future availability of the legacy wizards. If necessary, I can comment a few example files (with fake prices, obviously) so anyone can mess around with them.
Any help is greatly appreciated. Thank you!
If you can use .xlsm files, then you should use ADODB for connecting to the database.
Here is a good tutorial:
http://www.onlinepclearning.com/import-from-access-to-excel/
You can execute SQL statments to modify the database.
However if your database is password protected you will need to include that in the connection string,
and since it is very easy to crack protected vba projects, the password will not be safe at all.
I am building an Access 2010 db which will store and query information relating to time spend by users in our team. Part of the reporting needs to include whether timesheets have been submitted on time.
The process is currently being managed in Excel but is becoming cumbersome due to the growing size of the consolidated data. In the current process, the flag on whether someone is late with their timesheet is applied manually.
Instead of manually adding a Yes / No value to the excel data, I wondered whether it was possible to set up separate TransferSpreadsheet processes in Access to upload the excel data (and attach them to separate command buttons) such that, depending on which one is executed, the import process adds a Yes or a No value to the last column in the data as it's being uploaded.
That way we can import the excel data for those who submitted their timesheets on time (and 'stamp' them Yes for being on time) and then any subsequently late submitted timesheet data can be imported later (and 'stamped' with a No).
I have spent several hours looking at online forums and instruction pages but cannot find anything close to what I am trying to achieve, hence the reason for posting this here.
This is just one of the options I am considering but my VBA skills are insufficient to establish whether such a process could be handled in VBA. All help appreciated. Thanks.
Solved this one myself with a bit of perseverance. Ended up running a few DoCmd.RunSQL commands to Alter / Delete / Insert the tables I had and used a 'join' table to load the data from excel and then ran a command to append the data from the 'join' table to the main table. I just invoke slightly different commands to update the table field based on whether the data has been submitted late or on time.
I work with VBA in MS Access databases. I'd like to be able to log when files are saved, modified or deleted without having to update the existing code to do the logging when the pertinent events take place. I want the time, location and the name of the file.
I found a good example here: when file modified
However, it only allows for monitoring a particular location (path). I want to be able to log regardless of where the save, modify or delete takes place. I'm only allowed to program in the MS Office environment in this situation. It seems as though using the Windows API is going to be how this task will be achieved. However, I don't have much experience working with the API. Is there an easier way to achieve what I want that doesn't involve using the API?
Have you worked with After_Updates or After_Insert macros? Also, is your application split? Meaning there's a front-end and a back-end of the database. You can create a separate table that mirrors that table that you need to track changes for. Every time a table is updates, run a macro that inserts a row to that table.
I'm assuming you're saving files to the database. If that's the case, add a after_update or after_insert macro that can keep track of when then files are being modified or added to the table.
I'm trying to load data from my database into an excel file of a standard template. The package is ready and it's running, throwing a couple of validation warnings stating that truncation may occur because my template has fields of a slightly smaller size than the DB columns i've matched them to.
However, no data is getting populated to my excel sheet.
No errors are reported, and when I click preview for my OLE DB source, it's showing me rows of results. None of these are getting populated into my excel sheet though.
You should first make sure that you have data coming through the pipeline. In the arrow connecting your Source task to Destination task (I'm assuming you don't have any steps between), double click and you'll open the Data Flow Path Editor. Click on Data Viewer, then Add and click OK. That will allow you to see what is moving through the pipeline.
Something to consider with Excel is that is prefers Unicode data types to Non-Unicode. Chances are you have a database collation that is Non-Unicode, so you might have to convert the values in a Data Conversion task.
ALSO, you may need to force the package to execute in 32bit runtime. The VS application develops in a 32bit environment, so the drivers you have visibility to are 32bit. If there is no 64bit equivalent, it will break when you try and run the package. Right click on your project and click Properties and under the Debug menu you'll need to change the setting Run64BitRuntime to FALSE.
you dont provide much informatiom. Add a Data View between your source and your excel destination to see if data is passing through. Do do it, just double click the data flow path, select data view and then add a grid.
Run your app. If you see data, provide more details so we can help you
Couple of questions that may lead to an answer:
Have you checked that data is actually passed through the SSIS package at run time?
Have you double checked your mapping?
Try converting within the package so you don't have the truncation issue
If you add some more details about what you're running, I may be able do give a better answer.
EDIT: Considering what you wrote in your comment, I'd defiantly try the third option. Let us know if this doesn't solve the problem.
Just as an assist for anyone else running into this - I had a similar issue and beat my head against the wall for a long time before I found out what was going on. My export WAS writing data to the file, but because I was using a template file as the destination, and that template file had previous data that had been deleted, the process was appending the data BELOW the previously used rows. So, I was writing out three lines of data, for example, but the data did not start until row 344!!!
The solution was to select the entire spreadsheet in my template file, and delete every bit of it so that I had a completely clean sheet to begin with. I then added my header lines to the clean sheet and saved it. Then I ran the data flow task and...ta-daa!!! Perfect export!
Hopefully this will help some poor soul who runs into this same issue in the future!
I have a macro-enabled form that is intended to allow of a team of data-entry staff to record their daily efforts by transaction number. As forms go, it's pretty basic. However, I need to write a macro for a button that will let each person submit their records to a master sheet at the end of their shifts. I need to copy the range data and paste to a master workbook, with the person's name and the date being added to the individual rows.
I'm not sure how to facilitate the copy and paste to a closed workbook, or how to prevent problems with multiple people submitting to the form at the same time. Can anyone offer any suggestions please?
Make it update a database and then generate the report at the end of the day from that. I would also recommend that it inserts into the database each time the user inputs a record so that in the case of a power outage, all of their work from the day doesn't get lost. This will likely reduce the amount of concurrency issues too as the users will be periodically adding records instead of many records at the same time. Search 'VBA DAO' or 'VBA ADODB' to find examples on how to connect to a database with VBA.
You can do this simply by opening the workbook, inserting, and then closing the workbook. There is no simple way to insert into a closed workbook. Note, you could keep things hidden if you're trying to hide things from the user.
Add a reference to the "Microsoft Scripting Runtime" to get access to the filesystem and then use a simple file semaphore to control access to the common workbook.
Regarding the closed workbook, you use Application.Workbooks.Open(...) and .Close
Primary Choice would be to send the items to a database. Since thats already ruled out, I would suggest you write the data to a plain old .csv file. This will be easier to implement, and will not be limited by excel row limits.