I am currently writing some software in visual studio to analyse large amounts of data from an Access database using SQL. I have code to make a new calculated variable but am struggling with the amount of time it takes to write the data back into Access.
I am currently using some vb com code to communicate with my Access Database which is running in 2002/3 comparability mode. The following is my current code which runs a function in a loop to write to the database.
cnnOLEDB = New OleDbConnection
cnnOLEDB.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & DataDirectoryName & DatabaseFileName
cnnOLEDB.Open()
'cmdOLEDB = New OleDbCommand
cmdOLEDB.Connection = cnnOLEDB
ColumnString = "ID_VAR, ID_PAR, TimeValue, strValue, ID_UPL"
For RecordCounter = 0 To CalcData.GetLength(1) - 1
Var_ID = Var_ID + 1
ValueString = Format(Var_ID, "0") & ", " & Format(Parameter, "0") & ", #" & Date2String(CDate(CalcData(0, RecordCounter))) & "#, " & CalcData(CalcData.GetLength(0) - 1, RecordCounter) & ", " & Format(AsUpload, "0")
If DatabaseConnectionInsert("INSERT INTO " & TableName & " (" & ColumnString & ") VALUES (" & ValueString & ")", "Non-Query") = "Error" Then GoTo Close
Next
cnnOLEDB.Close()
Here is the Function:
Public Function DatabaseConnectioninsert(ByVal Query As String, ByVal Task As String) As String
'On Error GoTo Err
'If cnnOLEDB.State = ConnectionState.Open Then cnnOLEDB.Close()
cmdOLEDB.CommandText = Query
Select Case Task
Case "Read Recordset"
rdrOLEDB = cmdOLEDB.ExecuteReader()
DatabaseConnectioninsert = "Read Recordset"
Case "Read Scalar"
DatabaseConnectioninsert = cmdOLEDB.ExecuteScalar
Case "Non-Query"
cmdOLEDB.ExecuteNonQuery()
DatabaseConnectioninsert = "Non-Query"
End Select
Exit Function
Err:
MsgBox("Database connection error.")
DatabaseConnectioninsert = "Error"
End Function
I am currently trying to insert ~4500 records into the Access Database for each Parameter which takes ~3minutes. However when the project goes live it will have to deal with over 100000 records per Parameter so it is no where near fast enough.
To solve this issue I am thinking of either updating my code to .net or creating a record set, so I can move all of the data in Access at once. Can anyone give me some advice as to which will have the greatest impact to improving the speed of the inserts. I am running visual studio 2005 and Access 2007, updating the database to 2007 rather than compatibility mode is possible but not ideal , however my current code can't access it.
Thank you for your help
Josh
As ridiculous as it sounds, the very best performance you will get on an Access database is using the ancient DAO COM library. Use a RecordSet object to add the records one at a time in a loop and reference the fields by their index (ordinal position) rather than their names. You will find it much, much quicker than using oleDB.ExecuteNonQuery.
See the solution given here for more information. It's C# but it's easy enough to follow and convert to VB.NET if you want to try it out.
Edit
In deference to Remou's comments below: it would appear that Microsoft have in fact been keeping DAO technology up to date – in spite of declaring it obsolete back in 2002 – but you have to use the Office Access Redistributable rather than the better known DAO 3.6 library.
Related
I'm developing a desktop application for my company (it's just a prototype, actually) for bringing simple services to production department (tool and process instructions, presence records, product traceability, etc).
I'm using VB.net and wpf; some MS Access databases are the backend. It uses Access Database Engine 2016.
I have a simple function used to update database records, here's it:
Public Function UpdateDatabaseRecord(DatabaseNumber As Integer, tblName As String, SearchInColumn As String, SearchBy As String, UpdateColumn As String, UpdateTo As String) As Integer
Try
Using ServerConnection As New OleDbConnection(ConnectionSelector(DatabaseNumber))
Dim CommandString As String
If SearchInColumn <> "ID" Then
CommandString = "UPDATE [" & tblName & "] SET [" & UpdateColumn & "]=#UpdateColumn WHERE [" & SearchInColumn & "] = " & DQ & SearchBy & DQ
Else
CommandString = "UPDATE [" & tblName & "] SET [" & UpdateColumn & "]=#UpdateColumn WHERE [" & SearchInColumn & "] = " & SearchBy
End If
Dim CommandUpdate As New OleDbCommand(CommandString, ServerConnection) With {.CommandTimeout = 10000}
With CommandUpdate.Parameters
.AddWithValue("#UpdateColumn", UpdateTo)
End With
ServerConnection.Open()
CommandUpdate.ExecuteNonQuery()
ServerConnection.Close()
End Using
UpdateDatabaseRecord = 0
SetCentralTimeStamp(DatabaseNumber, tblName)
Catch ex As Exception
Msg(Reflection.MethodBase.GetCurrentMethod.Name & NewLine & ex.Message & NewLine & "Err. Number " & Err.Number, "Database Error", MessageBoxButton.OK, MessageBoxImage.Error)
UpdateDatabaseRecord = Err.Number
End Try
End Function
Sometimes the application uses this function in the main thread; sometimes it runs it from scheduled task (clocked by a forms.timer every two minutes). This tasks aren't obviously in the main thread, but the code has been used succesfully for a couple of months with computers directly connected to company network via LAN cable. Recently I tried my application on a laptop connected via wifi to the company network, and I get an exception that's crashing my app. I installed VS on that laptop to debug the code, and I found that an AccessViolationException is raised on line "CommandUpdate.ExecuteNonQuery".
I tried to google it but it seems that this exception shouldn't be raised from managed code.
The error occours only with laptop connected via wifi.
Also, another function reads data from the databases: sometimes I get the same error in it, in the line where the connection is being opened (connection.open).
What should I check?
Thanks in advance.
ps "ConnectionSelector" is a simple function which returns the appropriate ConnectionString based on database number.
EDIT 1 - The WiFi network is created plugging directly a LAN cable from the actual wired network, using a common wireless router. IP addresses are automatically assigned and the connection works, since the app is able to get data from databases.
I solved the problem myself and I'm posting here the solution I found: I hope it will help someone in the future.
I resolved simply enabling all OLEDB Services for the database connection. It looks like connection pooling really help stability and performance, especially when using a wifi network like I'm forced to do. If you want, Google "OLEDB Services" and look for enabling them all.
I am building a quite complex UserForm that uses ADO connection to connect to another Excel workbook that serves as a database and retrieve & insert data via SQL queries. Please note I am not allowed to use Access in this case.
I have already figured out how to use SELECT, but there is one particular error with INSERT I can't resolve. That bothers me a lot, I've put a lot of work to it.
First the connection (I use JET for retrieving data and ACE for saving data as I was not able to get JET to work for that):
Public Sub InsertDataToSheet(SQLCmd As String)
Dim cnx As Object
Set cnx = CreateObject("ADODB.Connection")
cnx.Open "Provider=Microsoft.ACE.OLEDB.12.0; Data Source='" & ThisWorkbook.Path & "\Database.xls'; Extended Properties=Excel 12.0;"
cnx.Execute SQLCmd
cnx.Close
End Sub
Then there is a subroutine linked to a Submit button that actually generates the Query as I need to save only filled out Textboxes and Combos to avoid Nulls:
Private Sub SaveRecord()
Dim SQL As String
SQL = "INSERT INTO [Report$A2:AM50000] ("
Dim i As Control
For Each i In Me.controls
If TypeName(i) = "TextBox" Or TypeName(i) = "ComboBox" Then
If i <> e Then SQL = SQL & i.Name & ","
End If
Next i
SQL = Mid(SQL, 1, Len(SQL) - 1) & ") VALUES(" ' Remove last space & comma
Dim j As Control
For Each j In Me.controls
If TypeName(j) = "TextBox" Or TypeName(j) = "ComboBox" Then
If j <> e Then
If j = "Unknown" Then MsgBox "Fire"
Select Case IsNumeric(j)
Case False
SQL = SQL & "'" & j & "'" ' Add single quotes around strings
Case True
SQL = SQL & j
End Select
SQL = SQL & ","
End If
End If
Next j
SQL = Mid(SQL, 1, Len(SQL) - 1) & ")" ' Remove last comma
' Connect
InsertDataToSheet (SQL)
End Sub
There are two particular textboxes in the form that work exactly the same. Normally, users enter numbers to them and everything saves fine (don't mind the '+' buttons):
Sometimes, however, users do not know the values but can't leave those empty. That's when they are supposed to tick the checkboxes to set the value(s) to 'Unknown':
Now there comes the funny part – for the Batch field, it saves fine. But when I set the Shipment ID to 'Unknown' (or any other string), it throws an error:
Note the fields are not Disabled, just Locked with some appearance changes. I was also unable to find any specific details about the error, but it seems there is some problem with the query:
(It says something like 'Incompatible data types in the expression'). The generated query is this:
Any ideas what goes wrong? I'd very much like to keep the functionality as it is know and solve the error rather than redesign it as it already took some effort and the fields can't stay empty.
Never used sql in xls workbooks, but I had this problem with SQL server already. There's nothing "wrong" with your query, the problem is that data type that's accepted on the field of the table you want to insert. Try to turn that field to use text values instead of numbers and it should work.
I created a new table in SQL server using SQL Server Management Studio but the MS Access front end linked to the database was not updated.
I tried reopening Access but still the new tables cannot be found. Yet when I check the SQL Server database they are there.
My tables in Access are linked to the database so I assumed any table or changes made in the SQL server database would be reflected in the Access front end. When I run a query in Access looking for the tables nothing is found. Another bit of information is when I right click and press view dependencies it says unable to view dependencies because
"unable to cast object of type 'System.DBNull' to type
'System.string'"
Something is maybe wrong with the way i save the query but I am not sure.
Your assumption:
I assumed any table or changes made in the SQL server database would
be reflected in the Access front end
...is not correct. Access does not automatically re-link when the SQL Server's schema changes, and it really can't. You're expecting Access assumes the data models between SQL Server and Access are the same. Even if your table and column names are exactly the same there are still differences to deal with since the data types have some differences. So, even in the best-case scenario Access does not have enough info to automatically re-link.
When you modify the SQL Server db you have to re-link from Access. Here's an article with some code that will allow you to do that quickly but note that you still have to launch it manually. And beware, as mentioned above, linking isn't that straightforward. If you use an automated method for linking the process will have to make some decisions, some of which make take you by surprise.
I have found the management of linked tables in access to be administratively tedious. In order to make my life simpler I have used the functions below that can be called to update the linked tables in access. This will take care of updating the structure of any changed table in SQL. Adding values to the SetTableNames function will bring in new tables
Private mstrTableNames(100) As String
Private const gcSQLDB as string = "MySQLServer"
Private const gcUserID as string = "SQLUserName"
Private const gcUserPassword as string = "SQLPassword"
Private const gcLiveDSN as string = "DSN"
Private const gcEmpty as string = ""
Public Function LinkLiveTables() As Boolean
Dim tdfLinked As TableDef
Dim strConnect As String
Dim intLoop As Integer
'Remove all non system tables from the application:
' !!!NB Add other exclusions so as to not delete tables that are not linked!!!
For Each tdfLinked In CurrentDb.TableDefs
If Left(tdfLinked.Name, 2) <> "MS" Then
If Left(tdfLinked.Name, 7) <> "tblTemp" Then
CurrentDb.TableDefs.Delete tdfLinked.Name
End If
End If
Next
'Create a linked table that points to SQL Server
strConnect = "ODBC;DATABASE=" & gcSQLDB & ";UID=" & gcUserID & _
";PWD=" & gcUserPassword & ";DSN=" & gcLiveDSN
SetTablesNames
For intLoop = 1 To 100
If mstrTableNames(intLoop) = gcEmpty Then GoTo ProcExit
Set tdfLinked = CurrentDb.CreateTableDef(mstrTableNames(intLoop))
With tdfLinked
.Connect = strConnect
.SourceTableName = "dbo." & mstrTableNames(intLoop)
End With
CurrentDb.TableDefs.Append tdfLinked
Next
ProcExit:
MsgBox "Connection to the LIVE tables was successful.", vbInformation
Exit Function
ProcError:
MsgBox "Link to LIVE tables Failed." & vbCrLf & vbCrLf & _
"Error Number : " & Err.number & vbCrLf & _
"Error Description : " & Err.Description, vbCritical
End Function
Private Sub SetTablesNames()
mstrTableNames(1) = "tblMoistureHist"
mstrTableNames(2) = "tblRawMaterials"
' ... add the additional table that you need as mstrTableNames(n) = "tablename"
End Sub
I have several excel files that are used for entering data. Files are identical in functionality, one for each service center of ours. In the form there is button that launches a macro which transforms data to table format on another sheet which is then later uploaded to Access db.
Everything worked fine on my own computer. Adding new rows, updating existing rows and deleting existing roles. I had used early binding which lead to problems when I moved files to our network drive. I managed to convert files to late binding but then other problems arose.
Most of the time, uploading to Access isn't working, especially when multiple users try to do stuff at the same time. Most common error code is that I am not using updateable query or that this method doesn't support backwards scrolling. I sorry for not reporting actual error codes, but I can't replicate them at the moment.
My connection code is as follows, it is bit of a mix of copy paste code from different examples.
Opening the connection and other prestuff
Sub excel2access()
Const adUseClient = 3
Const adUseServer = 2
Const adLockOptimistic = 3
Const adOpenKeyset = 1
Const adOpenDynamic = 2
Dim oConn As Object
Dim cmd As Object
Dim rs As Object
Dim r As Long
Dim criteria As String
Dim Rng As Range
Set oConn = CreateObject("ADODB.Connection")
Set cmd = CreateObject("ADODB.Command")
oConn.Open "Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source= '" & Range("dbpath").Value & "\" & Range("dbfile").Value & "' ;"
Set rs = CreateObject("ADODB.Recordset")
rs.CursorLocation = adUseClient
rs.CursorType = adOpenStatic
rs.LockType = adLockOptimistic
rs.Open "Select * from need_rows WHERE service_center = '" & Range("scenter_name").Value & "'", oConn
r = 2 ' the start row in the worksheet
Sheets("data").Select
This following bit looks through data in excel sheet and tries to find match from recordset found for that service center. If match is not found new record is created and if match is found the old record is updated.
Do While Len(Range("A" & r).Formula) > 0
With rs
criteria = Range("D" & r).Value
.Find "identifier='" & criteria & "'"
If (.EOF = True) Or (.BOF = True) Then
.AddNew ' create a new record
.Fields("service_center") = Range("scenter_name").Value
.Fields("product_id") = Range("A" & r).Value
.Fields("quantity") = Range("B" & r).Value
.Fields("use_date") = Range("C" & r).Value
.Fields("identifier") = Range("D" & r).Value
.Fields("file_type") = Range("file_type").Value
.Fields("use_type") = Range("E" & r).Value
.Fields("updated_at") = Now
.Update
Else
If .Fields("quantity") <> Range("B" & r).Value Then
.Fields("quantity") = Range("B" & r).Value
.Fields("updated_at") = Now
.Update ' stores the new record
End If
End If
.MoveFirst
End With
r = r + 1
Loop
rs.Close
Set rs = Nothing
Set oConn = Nothing
MsgBox "Confirmation message"
End Sub
Edit: Based on link by barrowc I changed cursor type to adOpenStatic. I made a test with several users trying to upload data at the same time and everything worked perfectly. Until one user stayed in the file and spent quite a while editing data there and then tried to upload data to db and got following error message: https://dl.dropbox.com/u/3815482/vba_error.jpg
Again, I am back where I started from.
Also, I am open to feedback on my code in general as well.
I am using Office 2010.
Am I doing it wrong? All help is appreciated.
You are going to have a lot of issues with the database being locked by other users. This is for several reasons:
From what I can see you are not handling errors. Therefore if your script errors half way through the connection will be left open thus causing lock problems.
By the looks of this, the macros could potentially keep the connection open for a decent amount of time (assuming no errors).
Having created a lot of macros connecting to an MS Access database I can tell you straight up. You are going to have a lot of connection issues where the database is being locked by spreadsheets that someone has left open all day/night due to things such as not handling unexpected errors (the connection never closes).
Even once you fix the problems all you need is ONE person to be using the spreadsheet with the old code and they will continue to lock the database.
One massive problem is that if someone connects to the database when its already open by someone else I believe they inherit the connection type of the already opened database resulting in a daisy chain of write locks. You then need to make sure all connections are severed in order to reset the connection.
You have also not shown us how the data is put into the spreadsheet in the first place. Perhaps you are not correctly closing the connection and that could potentially be the reason why sometimes the database is locked.
There are many different things you could try to get around this:
Easiest would be to use MS Access Front End + MS Access Back End.
Instead of pressing this button and uploading the data through connection strings you could make it save the file in a folder which would then be processed by an ms access database that is sitting there watching the folder. This would mean that you upload script would be written in MS Access and just be processing the files. Wouldn't be as instantaneous as your current approach but all write connections would be coming from the same machine/user in this circumstance.
Persist with the current method: eventually you may get it to a stable state but it will be a lot of frustration and effort as determining the reason for a lock may not always be easy. You could at least look at who has the file locked at the time and work from there but as mentioned earlier they may not have been the cause of the lock. They may have just inherited the lock type.
Personally I like to use MS Excel to display MS Access data for users, but avoid like a plague getting it to update MS Access. However if I was to do this I would do it through the use of oConn.Execute commands not opening a recordset, comparing and slowing pushing as that would keep the connection open too long.
Sorry for the wall of text. Hope this information helps.
sounds to me like Jet isn't reliable enough for your environment. I frequently use SQL Server / Access Data Projects to consolidate information from multiple spreadsheets into a single database backend that doesn't barf when you add a half dozen users.
You could also try using action queries.
First I would try to update using (you may need to format the now value)
dim count as long
oConn.Execute "UPDATE need_rows SET quantity = " & Range("B" & r).Value & ", updated_at = #" & Now & "# WHERE service_center = '" & Range("scenter_name").Value & "' AND identifier='" & Range("D" & r).Value & "' AND quantity <> " & Range("B" & r).Value", count
If count is zero then no row was updated, so either there was no row to update or the quantity hasn't changed. Either way we can try to INSERT, which will fail in the latter case, but without causing problems.
if count = 0 then
count = oConn.Execute "INSERT ...", count
if count = 0 then
' quantity has not changed, so nothing to do
else
' new record inserted
end if
else
' record updated
end if
I have the following bit of code that I have used in the past to export a DataSet generated from the Stored Procedure 'HISTORICAL_COSTS' to Excel.
Dim c As Long = 1
For Each dc As DataColumn In Me.WOCostDataSet. & _
HISTORICAL_COSTS.Columns
.Cells(1, c).Value = dc.ColumnName.ToString
.Cells(1, c).Font.Bold = True
c += 1
Next
Dim i As Long = 2
For Each dr As DataRow In Me.WOCostDataSet.HISTORICAL_COSTS.Rows
c = 1
For Each dc As DataColumn In Me.WOCostDataSet. & _
HISTORICAL_COSTS.Columns
.Cells(i, c).Value = dr.Item(dc.ColumnName).ToString
c += 1
Next
i += 1
Next
I am trying to re-use this code an different but similar application, but, I am running into an issue. The previous use of this code was used on a static table in our dBase generated by the Stored Procedure. And while this basically remains the same for the new application, the requirements now require the stored procedure to have an input parameter to be entered by the user (through VB.net) prior to execution. For a little back-story, you can follow that completed process here - Injecting a Parameter into Stored Procedure.
The application itself does in fact return a fully populated dataset, and I'd like our users to have the ability to export that generated dataset to Excel. So, I set up your prototypical 'EXPORT ME' button to do start the dirty work.
Upon raising the event; Excel opens and only my column names are being reiterated throughout the sheet. But, and here is the problem, the cells representing the row data - are blank.
I have come to the conclusion (I do admit that I may be wrong in this assumption) that the rows are not being populated due to the fact that the Stored Procedure needs an input parameter to do it's thing, and without that parameter there isn't any data to return for each row. Basically meaning that my code just won't work for what I am trying to do.
If I am right in my assumptions, any ideas as to how I might get that parameter into my code above so that the rows will be properly generated.
If I am wrong, well, any input on what be wrong with my logic or the code itself would be greatly appreciated.
Thanks,
Jasoomian
Stan,
Here is the code that generates the dataset:
Try
Dim FBConn As FbConnection
Dim MyConnectionString As String
MyConnectionString = "datasource=" _
& MyServer & ";database=" _
& TextBox4.Text & ";user id=SYSDBA;password=" _
& MyPassword & ";initial catalog=;Charset=NONE"
FBConn = New FbConnection(MyConnectionString)
Dim FBCmd As New FbCommand("HISTORICAL_COSTS", FBConn)
FBCmd.CommandType = CommandType.StoredProcedure
FBCmd.Parameters.Add("#I_PN", FbDbType.VarChar, 40)
FBCmd.Parameters("#I_PN").Value = TextBox1.Text.ToUpper
Dim FBadapter As New FbDataAdapter(FBCmd)
Dim dsResult As New DataSet
FBadapter.Fill(dsResult)
Me.HISTORICAL_COSTSDataGridView.DataSource = dsResult.Tables(0)
Dim RecordCount As Integer
RecordCount = Me.HISTORICAL_COSTSDataGridView.RowCount
Label4.Text = RecordCount
Catch ex As System.Exception
System.Windows.Forms.MessageBox.Show _
("There was an error in generating the DataStream, " & _
"please check the system credentials and try again. " & _
"If the problem persists, please contact your friendly " & _
"local IT department.")
End Try
Stackoverflow kindly suggests that I offer a bounty to anyone answering my question,but, since I don't have enough REP to create a sufficient bounty - does my ever-encompassing gratitude garner any coding love?
A few quick updates:
I tested my application by altering the Stored Procedure to inject the results into a new separate table, and then run my excel export against that table - and it works fine. But, since many people will be using the application at the same time, this is not a viable solution.
So, I am back to believing that there is an issue with this instance of the dataset needing a parameter to run correctly for the export.
Happy to answer any and all questions to the best of my ability.
You conclusion is definitely wrong -- once the dataset is populated the parameter is no longer a factor. I don't see anything obviously wrong with your code and, as you said, it worked elsewhere. I would start by setting a breakpoint and making sure that the dataset contains rows as you expect.