Increase Ms Access Insert Performance - sql

I am using MS Access 2010, split in front end / back end; on a network drive (WAN) with 16+ table with one table of users (1.3 Million) which is mostly used for user information and is not insert heavy and few other tables, which will receive upto 2000+ inserts daily.
I have been able to optimize most of the read/select queries. Although 1 chunk of my code looks as below. This can be used for upto 2000 iterations daily.
Do Until rec.EOF
Dim vSomeId As Integer
vSomeId = rec!SomeId
'StrSQL = StrSQL & "INSERT INTO TransportationDetails ( TransportationId, SomeId)" & _
'"VALUES(" & vTransportationId & ", " & vSomeId & ");"
StrSQL = "INSERT INTO TransportationDetails ( TransportationId, SomeId)" & _
"VALUES(" & vTransportationId & ", " & vSomeId & ");"
DoCmd.SetWarnings False
DoCmd.RunSQL (StrSQL)
DoCmd.SetWarnings True
rec.Edit
rec!SomeBoolean = rec!SomeOtherBoolean
rec.Update
rec.MoveNext
Loop
My objective here, is to reduce the number of calls to the db to insert all the values. and MS ACCESS does NOT support having more than 1 query in a statement, as I tried in the commented part of the code. I also think the recordset upate method is a lot time consuming, and if any one can suggest a better way to update the recordset.
Is there any way I can trick Access to insert & Update in less hits to db through SQL Queries, or any other access feature. Or optimize in anyway, It can take up to 30 mins some time. Decreasing it to At least 2 - 5 mins will be appropriate.
P.S.
I can not switch to SQL Server, It is JUST NOT POSSIBLE. I am aware it can be done in way more optimal way through sql server and Access shouldn't be used for WAN, but I don't have that option.
Solution:
I went with Andre's and Jorge's solution. The time decreased by 17 times. Although Albert's Answer is correct too as I found my main issue was with the sql statements in a loop. Changing the edits in the recordset to sql didnt impact much on the time factor.

I should point out that in the case of inserting rows, you will find FAR better performance by using a recordset. A SQL “action” query will ONLY perform better if you operating on a set of data. The instant you are inserting rows, then you don’t have a “set” insert, and using a DAO recordset will result in MUCH better performance (a factor of 10 to 100 times better).

If you have now
S = "SELECT SomeId, SomeBoolean, SomeOtherBoolean " & _
"FROM recTable WHERE someCriteria"
Set rec = DB.OpenRecordset(S)
change your statements into
"INSERT INTO TransportationDetails (TransportationId, SomeId) " & _
"SELECT " & vTransportationId & ", SomeId " & _
"FROM recTable WHERE someCriteria"
and
"UPDATE recTable SET SomeBoolean = SomeOtherBoolean WHERE someCriteria"
For performance, avoid looping over Recordsets where possible. Use SQL statements that operate on whole sets instead.

I recently ran into a problem where I have to import 200,000 records into 12 Access tables every 6 hours. This took too long as I was inserting each record one at a time.
I picked up a tip from a colleague who suggested using Linked Tables.
So I set up a linked table in my Access Database which was linked to a semi-colon delimited text file.
My program then created that semi-colon delimited text file every 6 hours.
You then select from the linked table into the table needed and that table is created.
This made my process immensely faster and I would definitely suggest it as an option.

Related

Faster alternative to DCount on SQL server-side

I want to check if some values are set to Non-Null when opening a record. Dependent on that, I will show a message box to the user.
Currently I use:
Dim blnValueExists As Boolean
If DCount("*", "dbo_tbl_Parts", "Part_Nr = '" & Me!txt_PartNr & "' AND [ValuesExist] is Null") Then
blnValueExists = False
Else
blnValueExists = True
End If
But this will do the operation on client side. How could I achieve the same by shifting the request to the server?
I have used DoCmd.RunSQL at other parts, but I do not know how to get a easy "true/false" result from SQL Server back without needing to use recordsets etc.
Using dcount() in this way, or even spending all the time and money to build server side SQL stored procedure?
ZERO difference in speed. While the dcount() function is a client side function, it creates the correct sql, sends to server, gets the value back.
As a result, there is no need to consider, build, or attempt to create any kind of say pass-though query to sql server.
Even if you migrate say a large table - 2 million rows.
Now, bind a form to that linked table.
Now, do this:
docmd.OpenForm "frmInvoices",,"InvoiceNumber = 121325"
Access will ONLY pull the one rocord down the network pipe, and dispite the form being bound directly to the table of 2 million rows?
Not all rows are pulled to the client side.
So, with your dcount(), there is no need to attempt some kind of server side SQL or even having to adopt some kind of stored procedure on the SQL server side.
Your current code is fine, and adopting a recordset to replace dcount() DOES NOT and WILL NOT run faster.
Try:
If CurrentDb.OpenRecordset("Select Top 1 Part_Nr From dbo_tbl_Parts Where Part_Nr = '" & Me!txt_PartNr.Value & "' And [ValuesExist] Is Null").RecordCount Then

MS Access: Method to process SQL without using RecordSource

I would like to be able to find the last record in a table easily for SQL INSERT INTO table statement. I was hoping there would be a MS Access object or function which could read a SQL statement without requerying the whole context just to find specific counts or records. As I have been programming the only code I know that reads SQL is a recordset, is there a dummy copy or source you could just read without repointing the record to another? Otherwise, I need a way to access the table in VBA and count all records with a method. If this is not code yet is should be, this would make it so easy to get around code dialects, other methods (unless you need to use form objects) if you know SQL.
I have tried several things, such as that cast a tbl variable but there would be a type mismatch.
This needs a last statement so I get the new record... I need to know how to get the last record in the table with a count.
CurrentDb.Execute "INSERT INTO FormsHelpTable ([ID], [HelpTitle], [Comment]) VALUES " & _
"(" & (lastRecTbl + 1) & ", '" & Me.Text53 & "', '" & Me.Comment & "')", dbFailOnError
Me.RecordSource = "SELECT * FROM FormsHelpTable"
Me.Requery
This worked to find the last record:
http://www.minnesotaithub.com/2013/08/count-records-vba-microsoft-access-2010/
Great code too.

Visual Basic.Net Insert multiple rows into a table

I am a relative newbie and trying to insert multiple rows (and data from textboxes) from one table into another and am stuck.
This SQL identifies the data to be inserted into the table
strsql = "SELECT '" & textbox1.text & "', '" & Textbox2.text & "', "
strsql = strsql & " a.TaskNum, a.StartDay, a.NumofDays FROM VETTimeLines as a"
strsql = strsql & " ORDER BY a.StartDay"
I started out along the lines of -> Insert into StudentProgram Values() code shown above, after 3 days of trying I now look forward to your advice.
Many thanks in anticipation
Peter
Inserting data using C# can be performed in variety of ways such as using Entity Framework or using ADO.NET, as you have chosen to do in this case.
Using ADO.NET you either write the insert as you did or by using DataAdapter approach. DataAdapter is capable of creating the SQL code for you, amongst other things. For example see:SQL DataAdapter. It is a good idea to not build a SQL string as you did because of sql injection threat as #Joel Coehoorn indicates in his comment above.
One way to overcome this is by using Parameters as shown in the above link. If you decide to provide the SQL for insert yourself with parameters, here is a good example:StackOverFlow-Insert using ADO.
The idea behind all of the above code is to create a connection object, create a parameter object, create a command object, open the database connection, execute the command and close the connection.
Try one of the above approaches and let's know your specific issue if any arises.

SQL bottleneck, how to fix

This is related to my previous thread: SQL Query takes about 10 - 20 minutes
However, I kinda figured out the problem. The problem (as described in the previous thread) is not the insert (while its still slow), the problem is looping through the data itself
Consider the following code:
Dim rs As DAO.Recordset
Dim sngStart As Single, sngEnd As Single
Dim sngElapsed As Single
Set rs = CurrentDb().QueryDefs("select-all").OpenRecordset
MsgBox "All records retreived"
sngStart = Timer
Do While Not rs.EOF
rs.MoveNext
Loop
sngEnd = Timer
sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time.
MsgBox ("The query took " & sngElapsed _
& " seconds to run.")
As you can see, this loop does NOTHING. You'd expect it to finish in seconds, however it takes about 857 seconds to run (or 15 minutes). I dont know why it is so slow. Maybe the lotusnotes sql driver?
any other ideas? (java based solution, any other solution)
What my goal is: To get all the data from remote server and insert into local access table
This document has some information about performance tuning in NotesSQL. If you aren't already, select your data from Notes Views instead of Notes Forms. NotesSQL will then leverage the indexes within the views for faster queries. You may need to create the view in the Notes database, but the performance benefit will make it worthwhile.
My recommendation is that you create a Pass-Through query that will get the data from the remote server. Then create a Make Table query that uses the aforementioned query as its source. Your function then would be simplified to a call to this second query.
The loop isn't doing "nothing" it's calling MoveNext, which is potentially doing A LOT.

Run Query Against ODBC Connected Table VBA

I have a table (readings) already connected by ODBC in Access that opens very quickly when I click on it.
However, when I try to run this in VBA I it locks up and never displays anything:
Dim strSql As String
strSql = "SELECT readings.ids " & _
"INTO ids_temp " & _
"FROM readings " & _
"WHERE readings.ids > 1234;" //This is id in middle of list
DoCmd.SetWarnings False
DoCmd.RunSQL strSql
DoCmd.SetWarnings True
For some reason this crashes the whole system. Any ideas?
Rather than using DoCmd, t's usually handled by your existing connection to create a Command object, which accepts SQL statements to use with the Command.Execute method.
Reading the documentation for DoCmd, it appears to primarily be intended for eexecuting Macros from the Access UI menus.
Does you Database have ids_temp table locally? If ids_temp table is Linked table it will delete the table, because select into CREATES NEW TABLE. If you want to add to table try INSERT INTO command. You can clean table before inserting the data.
So the error was actually my fault the id I was using was causing the Query to return about 6 million results. But, this method actually works great, I just create the table and link a list box on a different form to the table, then I just show the form. I do some closes and updates in between but overall it seems to work well. Thanks for the help
Let me say that DoCmd.RunSQL is never advisable, particularly with SetWarnings turned OFF, as you have no idea whether the result is what you expect or not. You've told VBA not to report errors, so you never know if all the records were inserted or not.
It's very easy to replace DoCmd.RunSQL with my SQLRun() function, posted here:
How can I get a value from the update query prompt in Access VBA?