This is related to my previous thread: SQL Query takes about 10 - 20 minutes
However, I kinda figured out the problem. The problem (as described in the previous thread) is not the insert (while its still slow), the problem is looping through the data itself
Consider the following code:
Dim rs As DAO.Recordset
Dim sngStart As Single, sngEnd As Single
Dim sngElapsed As Single
Set rs = CurrentDb().QueryDefs("select-all").OpenRecordset
MsgBox "All records retreived"
sngStart = Timer
Do While Not rs.EOF
rs.MoveNext
Loop
sngEnd = Timer
sngElapsed = Format(sngEnd - sngStart, "Fixed") ' Elapsed time.
MsgBox ("The query took " & sngElapsed _
& " seconds to run.")
As you can see, this loop does NOTHING. You'd expect it to finish in seconds, however it takes about 857 seconds to run (or 15 minutes). I dont know why it is so slow. Maybe the lotusnotes sql driver?
any other ideas? (java based solution, any other solution)
What my goal is: To get all the data from remote server and insert into local access table
This document has some information about performance tuning in NotesSQL. If you aren't already, select your data from Notes Views instead of Notes Forms. NotesSQL will then leverage the indexes within the views for faster queries. You may need to create the view in the Notes database, but the performance benefit will make it worthwhile.
My recommendation is that you create a Pass-Through query that will get the data from the remote server. Then create a Make Table query that uses the aforementioned query as its source. Your function then would be simplified to a call to this second query.
The loop isn't doing "nothing" it's calling MoveNext, which is potentially doing A LOT.
Related
I want to check if some values are set to Non-Null when opening a record. Dependent on that, I will show a message box to the user.
Currently I use:
Dim blnValueExists As Boolean
If DCount("*", "dbo_tbl_Parts", "Part_Nr = '" & Me!txt_PartNr & "' AND [ValuesExist] is Null") Then
blnValueExists = False
Else
blnValueExists = True
End If
But this will do the operation on client side. How could I achieve the same by shifting the request to the server?
I have used DoCmd.RunSQL at other parts, but I do not know how to get a easy "true/false" result from SQL Server back without needing to use recordsets etc.
Using dcount() in this way, or even spending all the time and money to build server side SQL stored procedure?
ZERO difference in speed. While the dcount() function is a client side function, it creates the correct sql, sends to server, gets the value back.
As a result, there is no need to consider, build, or attempt to create any kind of say pass-though query to sql server.
Even if you migrate say a large table - 2 million rows.
Now, bind a form to that linked table.
Now, do this:
docmd.OpenForm "frmInvoices",,"InvoiceNumber = 121325"
Access will ONLY pull the one rocord down the network pipe, and dispite the form being bound directly to the table of 2 million rows?
Not all rows are pulled to the client side.
So, with your dcount(), there is no need to attempt some kind of server side SQL or even having to adopt some kind of stored procedure on the SQL server side.
Your current code is fine, and adopting a recordset to replace dcount() DOES NOT and WILL NOT run faster.
Try:
If CurrentDb.OpenRecordset("Select Top 1 Part_Nr From dbo_tbl_Parts Where Part_Nr = '" & Me!txt_PartNr.Value & "' And [ValuesExist] Is Null").RecordCount Then
I am trying to write a VBA code in Microsoft Access that will create and run 60 queries to select new records from 60 linked tables and insert them into 60 tables of the same format. Some background may help here:
I have a large database (lets call the original database "A") that will eventually have over 60 tables, 60 forms & 60 reports. When one of our workers goes out into the field and doesn't have an internet connection, they are going to create new records on a copy of the database stored on their desktop (lets call the duplicate database "B"). Once they have an internet connection, I want them to be able to press a button on either database (I have been trying to code the macro on database A because I thought that would be easiest) and have the new records they created on database B inserted into database A.
I have found code online that seems to be just what I need but when I try to run the macro it gives Error 3022, which says:
The changes you requested to the table were not successful because they would create duplicate values in the index, primary key, or relationship
I have tried running this macro with both databases on my desktop with only 1 linked table with a primary key that is an autonumber, I tried running it with a random autonumber, I tried not having any primary key or index or autonumber and even no records at all. I even tried running it without any linked tables. All ways of trying give me the same Error 3022. I really don't want to create 60 queries one by one so any help would be greatly appreciated. Thank you wizards in advance :)
Here is the code I have tried:
Public Sub ImportTableData(ByVal pstrTable As String, ByVal pstrDb As String)
Dim strSql As String
strSql = "INSERT INTO " & pstrTable & vbNewLine & _
"SELECT *" & vbNewLine & _
"FROM " & pstrTable & " IN '" & pstrDb & "';"
CurrentDb.Execute strSql, dbFailOnError
End Sub
Public Sub ImportAllTables()
Const cstrDb As String = "C:\MyPath\DatabaseB.accdb"
Dim tdf As TableDef
Dim strMsg As String
On Error GoTo ErrorHandler
For Each tdf In CurrentDb.TableDefs
'ignore system and temporary tables '
If Not (tdf.Name Like "MSys*" Or tdf.Name Like "~*") Then
Call ImportTableData(tdf.Name, cstrDb)
End If
Next tdf
ExitHere:
On Error GoTo 0
Set tdf = Nothing
Exit Sub
ErrorHandler:
Select Case Err.Number
Case 3078
strMsg = "Input table " & tdf.Name & " not found."
MsgBox strMsg
Resume Next
Case Else
strMsg = "Error " & Err.Number & " (" & Err.Description & ") in procedure ImportAllTables"
MsgBox strMsg
GoTo ExitHere
End Select
End Sub
When I remove the piece of code: 'dbFailOnError', I get different errors. First thing that pops up when I run the macro is: ''Input table 'LocalTableName' not found'', which is the table I am trying to add the records to. Once I click 'Ok' on that pop up box, that is when Error 3134 pops up which says 'Syntax error in INSERT INTO statement'. I am assuming Error 3134 only pops up because it cannot find the local input table (the fist pop up box).
Also, I tried changing the line of code that says: 'Const cstrDb As String = ''C:\MyPath\DatabaseB.accdb''' to instead point to database A (which is the one I am coding the macro on) like this: 'Const cstrDb As String = ''C:\MyPath\DatabaseA.accdb'''. This doesn't give me the first pop up that says ''Input table 'LocalTableName' not found'' but it still gives Error 3134.
I have no idea what I am doing wrong and have spent over 20 hours on this problem trying dozens of different things. Any help would be greatly appreciated :)
Well, the main issue is how you going to ensure that the PK (primary keys) and FK (foreign keys) remain the same when they go out to the field and start entering data?
There is a good chance that PK/FK values will now be duplicated, or be the same for on one of the field users.
If a user out in the field adds a record, and someone at main work location adds record, they now are to very likly have the same PK value.
I suppose this might work if you use random autonumbers (never even knew that was a option after all these years!!!) - but I can't say even that going to be 100% reliable.
And when you import that copy from the out in the field user, then either:
You always ensure you accept and take the same PK value, or you let access generate a new PK - but if you do that, then the child records FK value would have to be updated then.
You not really try to do a simple import, but are attempting to do a database synchronization- a stunning and VERY advanced concept. And a very challenging problem.
Access (mdb format) did at one time support what is called database replication. This feature would be ideal for your setup.
However, but for newer accDB formats, it not supported anymore. (and quite sure by around access 2010, replication support was dropped anyway).
So, you could try random for the autonumber. I mean, you simple cannot have the PK's being duplicated on each computer - plain and simple.
The other possbile?
You add to each and every table that has a PK, and add a new column called PKF (f for in the field).
And for every table that has a FK, you add a new column called FKF (again, add F to this).
So, in the field, your PK/FK used is NOT the same as the main master database at work.
So, I wrote a Android sync routine based on above. The applcation was Access, but I moved the data to sql server (but same idea). I moved to sql server since my android phone could use its local database (sqlLite) and hit sql server. (but, it would be difficult to get android to hit some server and read + use a accDB file - but was easy to have Android phone hit the sql server database directly).
Gee, maybe they could use Android phones!!! but, this would assume you up to speed writing android software, 100% conformable with SQL server, and also access. I was lucky, had all 3 skill sets, so that is the road and hammer I choose.
And speaking of above?
Maybe your lucky, and you have sql server running at work (not express edition, but full edition). I suggest this considering, since the free edition of SQL express can be what we call a replication subscriber to a main sql database. This allows you to sync your local database with the main mothership database.
So, adopting free SQL server express on each field laptop could be a possible solution. Then when they get to a working network, they sync the database using replication.
But, you could try and roll your own sync system.
I did that for an android applcation I wrote, and for a desktop Access application I had. (but, to make all the moving parts easy, I did adopt sql server for database - continued to use Access as the application/UI part).
Now, using "random" for the PK looks to be a possible solution. I just don't know how random, and if this choice can reliable avoid PK collisions for autonumbers.
Random seems like the best road - but ONLY if that choice would prevent duplicate PK id's being used out in the field for new records.
Edit: Random - not even close - it not random enough
So, a bit of research - no, random PK will not work, you still often wind up with collisions - so that idea is off the table.
I figured it out finally. Basically I changed the line of code that executes the SQL to debug.print. This showed me what was going on. The problem is I had linked the tables from the other database when they didn't need to be linked. The names of the local tables would be 'Table A' & the linked tables were 'Table A1'. So there would be a query generated for 'Table A' and another query generated for 'Table A1'. Since there are no tables by the name 'Table A1' in Database B, the query wouldn't work. Plus the fact that, in the line of code executing the SQL, there was an option that says 'dbFailOnError' so since half of the queries weren't working, this option rolls back any updates made by the queries that did work.
I removed all of the linked tables and the macro runs perfectly, unless there are records that are the same on both databases. If I remove 'dbFailOnError' from the code, then the macro runs well no matter what.
So the macro is doing what I want it to but I would like to keep the 'dbFailOnError' part of the code so I will have to do 2 things. First, I have to solve the problem with the primary keys. The answer on this thread by Albert describes this problem well. Second, I have to adjust the SQL to be able to only select records that don't already exist in Database A. I am assuming I can do this by adding a WHERE to the end of the SQL.
I will make an update once I fix these problems, or if I just run an SQL server instead.
Thank you everyone for your help :)
I'm inserting data from one database to another, so I have 2 connections (Conn1 and Conn2). Below is the code (using pypyodbc).
import pypyodbc
Conn1_Query = "SELECT column FROM Table"
Conn1_Cursor.execute(Conn1_Query)
Conn1_Data = Conn1_Cursor.fetchall()
Conn1_array = []
for row in Conn1_Data:
Conn1_array.append(row)
The above part runs very quickly.
stmt = "INSERT INTO TABLE(column) values (?)"
Conn2_Cursor.executemany(stmt, Conn1_array)
Conn2.commit()
This part is extremely slow. I've also tried to do a for loop to insert each row at a time using cursor.execute, but that is also very slow. What am I doing wrong and is there anything I can do to speed it up? Thanks for taking a look.
Thought I should also add that the Conn1 data is only ~50k rows. I also have some more setup code at the beginning that I didn't include because it's not pertinent to the question. It takes about 15 minutes to insert. As a comparison, it takes about 25 seconds to write the output to a csv file.
Yes, executemany under pypyodbc sends separate INSERT statements for each row. It acts just the same as making individual execute calls in a loop. Given that pypyodbc is no longer under active development, that is unlikely to change.
However, if you are using a compatible driver like "ODBC Driver xx for SQL Server" and you switch to pyodbc then you can use its fast_executemany option to speed up the inserts significantly. See this answer for more details.
I know VB6 is a bit out of date but that's the application at the moment that I have inherited.
I have to do an update based on the results of an array to an access table.
The array contains a double and the id of the record to update.
The issue is that there are 120,000 records to update but on a test it took 60 seconds just to run it on 374 records.
This is my update code
Dim oCon As New ADODB.Connection
Dim oRs As New ADODB.Recordset
Dim string3 As String
oCon.Open "Driver={Microsoft Access Driver (*.mdb)};Dbq=" & App.Path &"\hhis.mdb;Pwd=/1245;"
oCon.BeginTrans
For i = 0 To maxnumberofrecords - 1
string3 = "UPDATE YourRatings SET yourRatings=" & YourTotalRatingsAll(i) & " Where thisID = " & thisID(i) & ";"
oRs.Open string3, oCon, adOpenStatic, adLockOptimistic
Next i
oCon.CommitTrans
oCon.Close
I added the "CommitTrans" based on some other articles that I had read but that doesn't seem to increase the speed.
The other problem is that I am going to have to run another query to rank the highest(1) to lowest (374) and update the db again...although I can probably do something with the array to add that column at the same time.
This seems quite slow to me especially when other post mention 200000 records in 14 seconds.
Have I just missed something?
Would it be quicker to load from a text file?
Thank you in advance for any help.
Mal
With Open you always constructing a new ResultSet object. Try oCon.execute string3 which only sends the SQL to your database without ResultSet overhead.
Ensure that you have an index on thisID.
Maybe your Access DB sits on a network drive. That could have a large performance impact. Try it local.
Why are you using the creaky old Access Desktop ODBC Driver instead of the Jet 4.0 OLEDB Provider?
Why are you opening a Recordset to do an UPDATE instead of calling Execute on the Connection?
Is there any reason you can't open the database for exclusive access to perform this operation? Locking overhead is all but eliminated and "cargo cult" techniques like thrashing with BeginTrans/CommitTrans lose any significance.
Do you have an index on your thisID field in the table?
Please do move to the magic bullet .Net as soon as possible. Your programs will be even slower but we won't have to read all the whinging blaming VB6.
Just to add to Wumpz answer, you might want to try just adding the query directly into the Access database and calling this using parameters. This SO thread says far more about it.
Parameters are far more stable and less hackable than injection.
I'm monitoring a folder for Jpg files and need to process the incoming files.
I decode the filename to get all the information I want and insert into a table and then move the file to another folder.
The file name is already contains all the information I want. Eg.
2011--8-27_13:20:45_MyLocation_User1.jpg.
Now I'm using an Insert statement
Private Function InsertToDB(ByVal SourceFile As String, ByVal Date_Time As DateTime, ByVal Loc As String, ByVal User As String) As Boolean
Dim conn As SqlConnection = New SqlConnection(My.Settings.ConString)
Dim sSQL As String = "INSERT INTO StageTbl ...."
Dim cmd As SqlComman
cmd = New SqlCommand(sSQL, conn)
....Parameters Set ...
conn.Open()
cmd.ExecuteNonQuery()
conn.Close()
conn = Nothing
cmd = Nothing
End Function
The function will be called for every single file found.
Is this most efficient way? Looks like its is very slow. I need to process about 20~50 files/sec. Probably a stored procedure?
I need to do this as fast as possible. I guess bulk insert not applicable here.
Please help.
Bulk insert could be applicable here - do you need them to be in the DB instantly, or could you just build up the records in memory then push them into the database in batches?
Are you multi-threading as well - otherwise your end to end process could get behind.
Another solution would be to use message queues - pop a message into the queue for every file, then have a process (on a different thread) that is continually reading the queue and adding to the database.
There are several things you can do to optimize the speed of this process:
Don't open and close the connection for every insert. That alone will yield a (very) significant performance improvement (unless you were using connection pooling already).
You may gain performance if you disable autocommit and perform inserts in blocks, commiting the transaction after every N rows (100-1000 rows is a good number to try for a start).
Some DB systems provide a syntax to allow insertion of multiple rows in a single query. SQL Server doesn't but you may be interested on this: http://blog.sqlauthority.com/2007/06/08/sql-server-insert-multiple-records-using-one-insert-statement-use-of-union-all/
If there are many users/processes accessing this table, access can be slow depending on your transaction isolation level. In your case (20-50 inserts/sec) this shouldn't make a big difference. I don't recommend modifying this unless you understand well what you are doing: http://en.wikipedia.org/wiki/Isolation_%28database_systems%29 and http://technet.microsoft.com/es-es/library/ms173763.aspx .
I don't think a stored procedure will necessarily provide a big performance gain. You are only parsing/planning the insert 20-50 times per second. Use a stored procedure only if it fits well your development model. If all your other queries are in code, you can avoid it.
Ensure your bottleneck is the database (i.e. moving files is not taking a lot of time), but since the OS should be good at this, check the points above. If you find that moving files is your bottleneck, delaying or moving files in the background (another thread) can help to a certain extent.