I sometime extract plenty of data from a very old database (MS Access). The serialized output is stored as YAML files and these files are locally used by other scripts to speed up the process.
Sometimes we to an update of the local files by extracting the (possible) new data from the database. This extraction is quite long and I would like to avoid it if the content of the concerned tables is the same as the last extraction.
Is it possible to get a sort of signature of the state of a table, or part of the table ?
In other words this would help understanding my question:
signature = db.GetSignature('SELECT * FROM foo where bar = 1')
if local_foo.signature != signature:
local_foo = db.Extract('SELECT * FROM foo where bar = 1')
What solutions could I use?
Using triggers
If you have the luxury of controlling the insert/update/delete functionality of the original Access database, the best/safest solution would be to implement database triggers to enable tracking. That way you could easily at least store a "last modified" value or keep a table that is responsible for storing extensive tracking information.
Unfortunately Access doesn't support triggers (unless you're using 2010+, see below), but you could implement triggers using VBA in the database.
Access 2010 introduced data macros, but I don't think that's an option here!
Using the scripting language
If you can't use database triggers, perhaps you could use a workflow like this:
Execute query and get entire result (single collection)
Turn the query result/collection into a JSON string (e.g. json.dumps() in Python)
Get a hash of the JSON string (e.g. hashlib.sha1() & hashObject.hexdigest() in Python)
Compare hash against the last stored hash of the query result.
Using VBA
To keep things database-side (to avoid transferring data), it could be useful to try generating the hash using VBA in the Access database.
You could use the hashing algorithm code mentioned in this SO post: https://stackoverflow.com/a/14749855
Example:
Using this SHA1 code: https://gist.github.com/anonymous/573a875dac68a4af560d
Dim d As DAO.Database
Dim r As DAO.Recordset
Dim s As String
Set d = CurrentDb()
Set r = d.OpenRecordset("SELECT foo, bar, baz FROM foobar")
s = ""
While Not r.EOF
s = s & r!foo & "," & r!bar & "," & r!baz & ";"
r.MoveNext
Wend
r.Close
Set r = Nothing
Set d = Nothing
s = SHA1TRUNC(s)
Related
i try to connect my xls with access database. Below code work greate when i have installed full access program on my machine. Problem is when i try tu use it on machine what have only installed Run-time version of access.
I have use this references:
Visual Basic For Applications
Microsoft Excel 14.0 Object Library
OLE Automation
Microsoft Office 14.0 Object Library
Microsoft Forms 2.0 Object Library
When i try to run below code i get error: ActiveX component can't create object or return reference to this object (Error 429)
Sub mcGetPromoFromDB()
Application.ScreenUpdating = False
Dim daoDB As DAO.Database
Dim daoQueryDef As DAO.QueryDef
Dim daoRcd As DAO.Recordset
'Error on line below
Set daoDB = Application.DBEngine.OpenDatabase("K:\DR04\Groups\Functional\DC_Magazyn\Sekcja_Kontroli_Magazynu\Layout\dbDDPiZ.accdb")
Set daoRcd = daoDB.OpenRecordset("kwPromoIDX", dbOpenDynaset)
Dim tempTab() As Variant
For Each Article In collecArticle
daoRcd.FindNext "IDX = " & Article.Index
Article.PromoName = daoRcd.Fields(1).Value
Article.PromoEnd = "T" & Format(daoRcd.Fields(2).Value, "ww", vbMonday, vbUseSystem)
Next
Application.ScreenUpdating = True
End Sub
Is IDX an indexed field?
Is kwPromoIDX optimized for this specific purpose? I mean does it only contain the fields required for this update, or are you pulling extra useless fields? Perhaps something like "SELECT IDX, [Field1Name], [Field2Name] FROM kwPromoIDX" would be more efficient.
Since you are only reading the table records, and don't seem to need to actually edit them instead of dbOpenDynaset, use dbOpenSnapshot.
Just throwing out an idea here, you'd have to test to see if it made any difference, but perhaps you could try to reverse your logic. Loop through the recordset 1 by 1 and locate the IDX within your worksheet.
Another thing I've done in the past is use .CopyFromRecordset and copied the entire recordset into a temporary worksheet and done the juggling back and forth entirely within Excel, to eliminate the back and forth.
Lastly, another approach can be to quickly loop through the entire recordset and populate an array, collection, ... and then work with it instead of Access. This way the data is all virtual and you reduce the back and forth with Access.
You'll need to do some testing to see what works best in your situation.
I am updating someone elses old code from a VB6 windows application to a VB.Net web application. I understand the old code, but I am not sure how I would translate it to .Net recordsets.
myStoredProcedure = "sp_WritevPayrollCurrent"
myCurrentPast = "'N'"
myStoredProcedure = "sp_ObtainSubClassID"
myClassName = "Payroll Major"
mySubClassName = "New Hire"
Set rs = TgetReadResultsetWithParms(myClassName, mySubClassName, (myStoredProcedure))
Also, I am not sure what will happen with "myStoredProcedure" being declared twice or could that be an error?
The TgetReadResultsetWithParms function is as follows (some Cases redacted to free up space):
Dim en As rdoEnvironment
Dim cn As rdoConnection
Dim rq As rdoQuery
rdoDefaultCursorDriver = rdUseServer
'open a connection to the tstdbexecutive database using DSNless connections
Set en = rdoEnvironments(0)
Set cn = connectionstring stuff here
Select Case myStoredProcedure
'create reusable rdoQuery and Call SQL server stored procedure.
Case "sp_ObtainClassID"
Set rq = cn.CreateQuery("", "{call " & cstDatabaseName & ".dbo.sp_ObtainClassID(?)}")
Case Else
Set rq = cn.CreateQuery("", "{call " & cstDatabaseName & ".dbo." & myStoredProcedure & "(?)}")
End Select
'set the input argument value for the store procedure.
rq(0) = myParm1
'open the Resultset and pass it back to calling procedure
Set TgetReadResultsetWithParm = rq.OpenResultset(rdOpenKeyset, rdConcurReadOnly)
The VB6 code is using Remote Data Objects. I think you will have to read the documentation to understand what the VB6 is doing, and then rewrite in VB.Net using ADO.Net to achieve the same functionality. AFAIK there is no handy cheat-sheet which shows how RDO compares with ADO.Net, unfortunately. There are some for ADO-ADO.Net.
Alternatively you could add a reference to Remote Data Objects in your .Net project - you can use COM objects in .Net - and then use the same code. If there is a large amount of the old code, this could be a pragmatic way to make the task more manageable, although it makes the code harder to understand for other programmers who would probably be used to ADO.Net. It's better to rewrite if you can afford to.
I am trying to store and retrieve data that is entered into Excel into Access. I am an Access newbie and already have an Excel program on the front end, leading me to drive the import from Excel. I have successfully figured out how to save my Excel Named Range into the desired Access table, but now I need to figure out how to pull specific data from that Access table back into Excel. I know I can simply use the Get External Data feature from Excel to import the entire Access table into Excel, but I need to be able to only import a specific portion of the table into Excel based upon a predetermined set of parameters. Is this possible to do?
As a background to the program, basically it stores data from part number runs. Not only do I need to save new part runs into an Access database (which I figured out), but I also need to be able to pull previous part number runs from Access back into Excel to perform further analysis. That is why I need to figure out how to import only a specific portion of the table. I'm not sure what code, if any, I can post since I'm basically looking for code from scratch; but if there's any code you think you need from my Excel program I'm happy to provide it. Thanks for your help. Below is the code:
Sub GetSpecData()
Application.ScreenUpdating = False
'*******************************************************************************
'Open the connection to the data source.
Set Connection = New ADODB.Connection
Connection.Open "Provider=Microsoft.Jet.OLEDB.4.0; " & "Data Source=\\Server2013\network_shared\QC SHARED\Databases\P&Q_Tracking_Data_Storage.mdb;"
'*******************************************************************************
'Create the new RecordSet.
Set Recordset = New ADODB.Recordset
With Recordset
'Define the appropriate Filter(s) and notify the user of the selection criteria.
Src = "Select * from Raw_Data where Tag = 'GHI' "
Src = Src & "or Tag = 'DEF' "
Src = Src & "or Tag = 'LMN'"
.Open Source:=Src, ActiveConnection:=Connection
'Write the field names.
For ODCol = 0 To .Fields.Count - 1
Tracking.Sheets("Selected Past Data").Range("B7").Offset(0, ODCol).Value = .Fields(ODCol).Name
Next
'Write the recordset.
Tracking.Sheets("Selected Past Data").Range("B7").Offset(1, 0).CopyFromRecordset Recordset
End With
Set Recordset = Nothing
Connection.Close
Set Connection = Nothing
'*******************************************************************************
'Create and format the table from the Recordset.
With Tracking.Sheets("Selected Past Data")
DataLastRow = .Range("A" & Rows.Count).End(xlUp).row
.ListObjects.Add(xlSrcRange, Range("B7:M" & DataLastRow), , xlYes).Name = "INC2tbl"
.ListObjects("INC2tbl").ShowTotals = True
End With
Application.ScreenUpdating = True
'*******************************************************************************
End Sub
After googling the suggestion in the comment below, I have a couple questions. First, the code above seems to filter access data by three keys: GHI, DEF, and LMN. Am I interpreting that correctly? Second, where it says "Select * from Raw_Data where Tag = 'GHI' ", since that's in quotes, that's not actual code that will be executed, correct? That's simply a prompt or something like it, correct?
SELECT * FROM TABLE; OUTPUT TO TABLEName.EXL
FORMAT sql;
First, the code above seems to filter access data by three
keys: GHI, DEF, and LMN. Am I interpreting that correctly?
Second,
where it says "Select * from Raw_Data where Tag = 'GHI' ", since
that's in quotes, that's not actual code that will be executed,
correct? That's simply a prompt or something like it, correct?
First and second questions relate closely, so I'll answer as one. The basic idea here is that there is a connection to the database you defined (in the Connection.Open statement). The connection itself does nothing then that; establish a connection. The actual communication with the database is done in a specific language, named SQL (there are many dialects, but for simplicity sake, call it SQL for now). So your code in in the VBA language, but the communication with the database is done in SQL. The VBA code has to produce SQL statements (and those are strings, so you need the quotes). That's what the Src variable holds; and SQL statement. In this case:
"Select * from Raw_Data where Tag = 'GHI' or Tag = 'DEF' or Tag = 'LMN'"
I think this is not the place to teach you SQL, but there is plenty information on the net about it. What this statement does is:
select all columns from table "Raw_Data", but only those rows of data that has 'GHI', 'DEF' or 'LMN' in the Tag column.
So to receive all the data from table "part_numbers", you would have to use:
"SELECT * FROM part_numbers;"
And if you need to import only columns "col_1" and "col_2", you would use:
"SELECT col_1,col_2 FROM part_numbers;"
And if you need to import only columns "col_1" and "col_2", and only rows with "part_id" lower than 1000, you would use:
"SELECT col_1,col_2 FROM part_numbers WHERE part_id < 1000;"
That's it really. Now if your recordset (think of it as an array that holds the data you asked for) has queried the database, you can loop it to parse the data. Something like:
Set Recordset = Connection.Execute(Src)
Do until Recordset.EOF
debug.print Recordset!col_1
Recordset.movenext
loop
Again, google something like: "vba excel adodb access", and you'll hit plenty examples.
After researching SQL code formatting more I have made more sense of the subject. Thanks for the help.
I'll start with the background story before explaining the problem with my code. I'm using MS Access 2010. I've been able to import a table of data with two columns. Then I was able to curate the data by adding fields with appropriate values to the imported table. Now, I need to take the curated table and integrate it into my data base. However, I cannot use any of Microsofts built in queries as none of these appear to be able to do what I need. The integration breaks the table apart, yes, but it needs to preserve the relationships of the data in each record.
To this end I've been writing some code in VBA:
Function IntegrateNIRData(curatedTable, queryRecords)
On Error GoTo Error_Handler
Dim db As DAO.Database
Dim rsCuratedTable, rsDBRecords As DAO.Recordset
Dim iCount As Integer
Set db = CurrentDb()
Set rsCuratedTable = db.OpenRecordset(curatedTable, dbOpenTable) 'open the recordset for use (table, Query, SQL Statement)
Set rsDBRecords = db.OpenRecordset("NIR_Samples_verify", dbOpenDynaset, dbExecDirect, dbOptimisticValue)
With rsCuratedTable
If Not (.BOF And .EOF) Then
Do While Not .EOF
' Rest of your code here.
rsDBRecords.AddNew
'Assign Fields here.
rsDBRecords![Product Name] = rsCuratedTable![productName]
rsDBRecords![Lot Number] = rsCuratedTable![lotNumber]
rsDBRecords!counts = rsCuratedTable![counts]
rsDBRecords![subsets] = rsCuratedTable![subsets]
rsDBRecords![Date Taken] = rsCuratedTable![dateTaken]
rsDBRecords.Update
rsDBRecords.Bookmark = rsDBRecords.LastModified
.MoveNext
Loop
End If
End With
rsCuratedTable.Close 'Close the recordset
rsDBRecords.Close 'Close the recordset
Error_Handler_Exit:
On Error Resume Next
'Cleanup after ourselves
Set rs = Nothing
Set db = Nothing
Exit Function
Error_Handler:
MsgBox "MS Access has generated the following error" & vbCrLf & vbCrLf & "Error Number: " & _
Err.Number & vbCrLf & "Error Source: IntegrateNIRData" & vbCrLf & "Error Description: " & _
Err.Description, vbCritical, "An Error has Occured!"
Resume Error_Handler_Exit
End Function
The Function hangs on this line, the second OpenRecordset:
Set rsDBRecords = db.OpenRecordset("NIR_Samples_verify", dbOpenDynaset, dbExecDirect, dbOptimisticValue)
To my understanding this may have something to do with Workspaces and the Jet engine not accepting a ms query that spans multiple tables. Of course, I could also be way off. Any advice at this point would be greatly appriciated.
Update:
Several of you have asked similar questions so I felt I should clarify the following:
1) NIR_Samples_verify is an MS access select query that generates a table of records from several of the tables in the database.
2) I keep getting two errors depending on what I set the RecordsetOptionEnum and LockTypeEnum to in the OpenRecordset method.
One is Error Number 3027 Database is Read-Only
Two is Error Number 3001 Invalid Arguement
3) To my understanding the rest of the code should be fine, it is just the OpenRecordset method that is causing the problem.
Update 2:
I am thinking that maybe access is not capable of doing what I would like. Let me illustrate. If I had two tables both with primary keys and these keys are referenced in a third table that links the two tables causing a many-to-many relationship, then the code would have to not only add the new data to the two tables, but also generate an appropriate record in the third table to maintain the relationship in the data. Hope that makes since. I do appriciate the help and experience.
Update 3:
Have been searching the net and found the following:
From this post it says the query is only updatable when:
1) It is based on a single table.
2) It is based on a query based on a single table.
3) It is based on a query based on tables with a one-to-one relationship.
Not knowing what the contents of NIR_Samples_verify are, I'd be highly suspicious of the dbExecDirect
From the help file
dbExecDirect
"Runs a query by skipping SQLPrepare and directly calling
SQLExecDirect (ODBCDirect workspaces only). Use this option only when
you’re not opening a Recordset based on a parameter query. For more
information, see the "Microsoft ODBC 3.0 Programmer’s Reference." "
I don't see you supplying any parameters.
-- Edit --
Typically I'll open a record set like this
Set rsDBRecords = db.OpenRecordset("select bar from foo where bar > 10", _
dbOpenDynaset, _
dbSeeChanges)
(Especially if I want to alter the data init)
Hopefully that'll move you further in your project.
-- Edit 2 --
It sounds like NIR_Samples_verify is to complicated to be edited. Actually, given that it is a join of multiple tables doing an Add on it doesn't make much sense, and Update MIGHT make sense in some cases.
Your ultimate solution is really going to be doing multiple Adds on multiple record sets (one for each table being referenced in NIR_Samples_verify); much like if you were entering the data into the DB by hand. You add the records that aren't dependant on anything else first (remembering to grab keys to use in the dependant tables).
As it turns out my hunch was correct. The problem had to do with MS Access having updatable and non-updatble queries (See my edits of the question). The main problem was not only does Microsoft not make this information apparent, but there is no master list on their site either. Thank you everyone for the help. Feel free to see this article for more details.
I need to open foxpro free tables in vb.net using the oledb connection.
But... I only need to get the column names. I don't really need to 'select' anything.
I am trying to dynamically browse through all our free tables and set up a listing of every column from every file and xref that to another free table that contains a description of each column.
I have a working model now, but it requires that I do...
SELECT TOP 1 FROM "File" ORDER BY 1
But on the largest table, it takes over two minutes just to read in the first record and there are over 250 tables. Overall, it takes between 15 and 20 minutes.
Or, is there another way to only get the first record of the table without using 'ORDER BY'?
Here's what I have so far. "File" is passed in as a parameter.
It would contain info like "C:\data\table1.dbf"
Dim filePath As String
filePath = IO.Path.GetDirectoryName(file)
myOledbConnection = New OleDbConnection("Provider=VFPOLEDB.1;Data Source=" & filePath & ";Collating Sequence=MACHINE")
myOledbCommand = New OleDbCommand
myOledbDataAdapter = New OleDbDataAdapter
Dim fields, from, order As String
fields = "select top 1 *"
from = " from " & file
order = " order by 1"
myOledbCommand.CommandText = fields & from & order
myOledbCommand.Connection = myOledbConnection
myOledbDataAdapter.SelectCommand = myOledbCommand
myOledbDataAdapter.Fill(dt)
I then take the datatable (dt) and loop through to get the column information.
I would like it to be as quick as Visual Studio is when I create a dataset and load all tables from the directory through the wizard. It is able to very quickly find all the column information without reading in the data from the table.
Let me know if you need more information.
Thanks.
Why do you need to get any records at all? You should be able to say:
SELECT * FROM "File" where 1 = 0
This will give you an empty result set, will also give you metadata on the projection returned.
You might also want to look into the GetOleDbSchemaTable method on the OleDbConnection class, as it will allow you to get information about the schema of the database without having to perform a query.
You can also use the Microsoft ADO Extensions for Data Definition Language and Security through COM interop (mxADOX.dll) to get the schema information as well.
I have not tried this/. But, it looks like the way to go.
Specifically the "GetSchema" method on OleDbConnection instance.
http://msdn.microsoft.com/en-us/library/ms254934(VS.80).aspx