Scenario
I have a database made up of 4 columns and 6 million rows.
I want to be able to be able to use Excel to query this data in the same way in which I can query it using the Sql Server Management Studio.
I've had a google around and read a few articles but in all honesty I dont really know enough about the implementation to really know what to look for in a tutorial.
Question
Could someone please explain to me the way in which this works and how I might go about implementing what seems like a fairly simple idea?
EDIT:
So for example -
I have a few simple queries that I want to run on this data.
How do I go about setting these up?
Is the idea that all of the data is first loaded into the excel sheet (or at least linked to the excel sheet so that the user can select the data to view on the fly?) - I cannot physically load in 6 million rows as Excel cannot take that level of data.....
Definitely don't pull all the data into Excel, that will bring it to its knees.
Normally, when I need an Excel (VBA) workbook to retrieve some data from a database, I will just have Excel connect to the database with the appropriate connection string, pass the SQL query string to the database, and retrieve the returned records. Below is some sample VBA code that retrieves some information from an MS Access database.
txtQueryString = "SELECT * "
& "FROM myDataTable " _
& "WHERE fld1 = '" & myCriteria & "'"
txtConnectString = "Provider=Microsoft.Jet.OLEDB.4.0;" _
& "Data Source=" & dbPathString & ";" _
& "Persist Security Info=False"
Dim rst As ADODB.Recordset
Set rst = New ADODB.Recordset
rst.Open txtQueryString, txtConnectString, adOpenKeyset, adLockReadOnly, adCmdText
myData1 = rst.Fields("fld1").Value
myData2 = rst.Fields("fld2").Value
.
.
.
rst.Close
Set rst = Nothing
The connection string to SQL Server will be different. I use this website for all my connection string needs.
Related
The data I am using is stored in a SQL database and an Access database. Regular data entry needs to be made in the Access database, however, I would like to write my queries in SQL Management Studio. I have connected the two databases using an ODBC connection and I can now view the tables in Management studio but the connection is not live, so any updates in Access are not reflected in the tables in Management Studio.
You can create a linked table from Access to SQL Server (INSERT INTO). You can use SQL or VBA to export all data from Access to SQL Server.
Here is one idea.
Sub modExportToPG()
Dim rs As DAO.Recordset
'We only want to export physical user defined visible and hidden tables
Set rs = CurrentDb.OpenRecordset("SELECT Name " & _
" FROM MSysObjects " & _
" WHERE Type=1 AND Flags < 9 ORDER BY Name;")
Do Until rs.EOF
'export the tables but export as lower case table names
DoCmd.TransferDatabase acExport, "ODBC Database" _
, "ODBC;DRIVER={PostgreSQL Unicode};DATABASE=mydb;SERVER=myserver;PORT=5432;UID=myuser;PWD=mypwd" _
, acTable, rs("Name"), LCase(rs("Name"))
rs.MoveNext
Loop
rs.Close
End Sub
Also, see this for more ideas.
https://www.mssqltips.com/sqlservertip/1480/configure-microsoft-access-linked-tables-with-a-sql-server-database/
Came upon this s little late but I am in the midst of working through a project where we update SQL side tables at the same time that we update local and shared (server side) Access tables. It is a fairly extensive application, but the SQL updates are relatively easy and there are options. The application integrates with a Point of Sale back office server and we are sensitive to the data that gets updated instantly, so those items get tucked into batches that require approval before they get "posted" to SQL. Those items that are not sensitive we post immediately using the afterupdate property for each field on the main form using the following code (sample only):
Private Sub Field.AfterUpdate
Dim Var1 as string
Dim Var2 as string
Dim Var3 as string
'Capture Before and after values for the field (as well as using the_
newvalue to post to SQL, we store these separately as audit records_
so we capture both old and new).
Var1 = Me.Field.OldValue
Var2 = Me.Field.NewValue
Var3 = Me.PID.Text
'Execute the SQL update
DoCmd.DbExecute "Update dbo.SQLTABLE SET SQLField = '" & Var2 & "'_
WHERE SQLTABLE.PID = '" & Var3 & "'
End Sub
Note: PID is a record id attached to every record in the access form that matches a corresponding record in a SQL Table (primary key).
It is a lot of work if you have a number of fields to post but it's as good as it gets using linked tables in my experience. This is a simple example and ours are far more extensive, but it does the trick.
There is a excel workbook which is used as a database kept on shared folder. The users have other workbook which has the userform which allows the insert and update into the database.
I am using connection.Execute method for insert and Update and use recordsaffected for my confirmation.
Although the recordsaffected gives the correct, sometimes insert or update never happens.
What can I do to fix this?
Set oCn = CreateObject("ADODB.Connection")
oCn.Open "Provider=Microsoft.ACE.OLEDB.12.0;" & _
"Data Source=" & shtSetting.Range("filePath").Value & ";" & _
"Extended Properties=Excel 12.0;"
strSQL = "Insert Into [Sheet1$] (" & _
"sr_no, process,employee_id,employee_name) Values ( " & _
"'" & sr_no & "','" & process & "','" & emp_id & "','" & emp_name & "')"
oCn.Execute strSQL, r_status
oCn.Close
Set oCn = Nothing
Unless there's a strong reason to store the data in an Excel spreadsheet, though, I'd consider offloading the database management portion of it to an Access database (even if you don't have Access, you can still create an mdb/accdb file). If nothing else, that's going to give you access to indexing, which can improve performance tremendously. You can still mirror the Access data in Excel by using data connections (or other techniques).
If you are stuck with doing everything in Excel, one (potentially slow) solution is to query the sheet once you've performed your update, to confirm that it has made the change that you expect. If you're doing an insert, you may be able to get away with just counting the records, but it's definitely safer to look for the exact value you just inserted.
Also: if the users are making multiple updates, you might want to consider keeping the connection open until the form closes. (Though I don't recall if this causes locking issues with an Excel backend or not. It would generally improve performance in Access.)
I am using MS Access 2010, split in front end / back end; on a network drive (WAN) with 16+ table with one table of users (1.3 Million) which is mostly used for user information and is not insert heavy and few other tables, which will receive upto 2000+ inserts daily.
I have been able to optimize most of the read/select queries. Although 1 chunk of my code looks as below. This can be used for upto 2000 iterations daily.
Do Until rec.EOF
Dim vSomeId As Integer
vSomeId = rec!SomeId
'StrSQL = StrSQL & "INSERT INTO TransportationDetails ( TransportationId, SomeId)" & _
'"VALUES(" & vTransportationId & ", " & vSomeId & ");"
StrSQL = "INSERT INTO TransportationDetails ( TransportationId, SomeId)" & _
"VALUES(" & vTransportationId & ", " & vSomeId & ");"
DoCmd.SetWarnings False
DoCmd.RunSQL (StrSQL)
DoCmd.SetWarnings True
rec.Edit
rec!SomeBoolean = rec!SomeOtherBoolean
rec.Update
rec.MoveNext
Loop
My objective here, is to reduce the number of calls to the db to insert all the values. and MS ACCESS does NOT support having more than 1 query in a statement, as I tried in the commented part of the code. I also think the recordset upate method is a lot time consuming, and if any one can suggest a better way to update the recordset.
Is there any way I can trick Access to insert & Update in less hits to db through SQL Queries, or any other access feature. Or optimize in anyway, It can take up to 30 mins some time. Decreasing it to At least 2 - 5 mins will be appropriate.
P.S.
I can not switch to SQL Server, It is JUST NOT POSSIBLE. I am aware it can be done in way more optimal way through sql server and Access shouldn't be used for WAN, but I don't have that option.
Solution:
I went with Andre's and Jorge's solution. The time decreased by 17 times. Although Albert's Answer is correct too as I found my main issue was with the sql statements in a loop. Changing the edits in the recordset to sql didnt impact much on the time factor.
I should point out that in the case of inserting rows, you will find FAR better performance by using a recordset. A SQL “action” query will ONLY perform better if you operating on a set of data. The instant you are inserting rows, then you don’t have a “set” insert, and using a DAO recordset will result in MUCH better performance (a factor of 10 to 100 times better).
If you have now
S = "SELECT SomeId, SomeBoolean, SomeOtherBoolean " & _
"FROM recTable WHERE someCriteria"
Set rec = DB.OpenRecordset(S)
change your statements into
"INSERT INTO TransportationDetails (TransportationId, SomeId) " & _
"SELECT " & vTransportationId & ", SomeId " & _
"FROM recTable WHERE someCriteria"
and
"UPDATE recTable SET SomeBoolean = SomeOtherBoolean WHERE someCriteria"
For performance, avoid looping over Recordsets where possible. Use SQL statements that operate on whole sets instead.
I recently ran into a problem where I have to import 200,000 records into 12 Access tables every 6 hours. This took too long as I was inserting each record one at a time.
I picked up a tip from a colleague who suggested using Linked Tables.
So I set up a linked table in my Access Database which was linked to a semi-colon delimited text file.
My program then created that semi-colon delimited text file every 6 hours.
You then select from the linked table into the table needed and that table is created.
This made my process immensely faster and I would definitely suggest it as an option.
I am trying to figure out how to download using an ADO/DAO connection in Access VBA to get the contents of a table from SQL server. I am trying to avoid using a linked table because the DB requires a password and I keep running into issues with getting it to not ask for the login info. Are there any ideas or references for me to start with on this matter?
It appears either way you'll need to provide SQL credentials.
There's more involved without linking a table, basically you'd want a recordset for the source and the "target" table to iterate over.
targetrs = CurrentDb.OpenRecordset("Target", dbOpenTable)
Dim Con As New ADODB.Connection
Dim sqlStr As String
Con.Open _
"Provider = sqloledb;" & _
"Data Source=SqlServer;" & _
"Initial Catalog=MyDB;" & _
"User ID=sa;" & _
"Password=p#ssW0rd;"
Dim rsSource As New ADODB.Recordset
rsSource.Open "select * from SOURCE", Con
do until rsSource.eof
targetrs.addnew
for each field in rsSource
targetrs.fields(field.Name) = rsSource.fields(field.Name)
next
targetrs.update
rssource.movenext
loop
Since you still have to have the credentials, you could dynamically link the table instead:
docmd.TransferDatabase acLink,"ODBC Database",
"ODBC;Driver={SQL Server};Server=MySQLServer;Database=MYSQLDB;
Uid=USER;Pwd=PASSWORD",acTable,"SQLtable","MyAccessTable"
Use of a linked table does not require you store or have the user password in that linked table.
If you execute a SINGLE logon at application startup then all linked tables will work.
Linked tables work WITHOUT a prompt for user or password.
Linked tables work WITHOUT you having to store the user ID or password in the link.
Access will cache the user name + password if you logon as per the instructions here:
http://blogs.office.com/b/microsoft-access/archive/2011/04/08/power-tip-improve-the-security-of-database-connections.aspx
So to download a table to a local, then you ONLY need this code:
For a new local table (create table query):
CurrentDb.Execute "SELECT * INTO LocalTableCreate FROM ServerTable"
Append to existing table:
CurrentDb.Execute "INSERT INTO LocalTable SELECT * FROM ServerTable"
And if some really strange reason and desire exists create and promote world poverty and do things the hard way like a turtle with time to waste and not use a linked table?
Well you could create a linked table via the “transfer database” command. It is only one extra line of code in front of the above code and then AGAIN the above two examples would work fine.
However I see little if any advantage to creating + deleting a linked table.
I suppose for reasons of performance or perhaps for security or the legitimate reason of you not knowing the table ahead of time? Then I would suggest you use a saved a pass-though query as performance will be even faster.
So you can use this code:
Dim qdfPass As DAO.QueryDef
Set qdfPass = CurrentDb.QueryDefs("MyPass")
qdfPass.SQL = "select * from dbo.MyTable;"
CurrentDb.Execute "INSERT INTO LocalTable SELECT * FROM MyPass”
Note that the sql used in above qerydef MUST be native T-SQL and can be a view or even a store procedure like:
qdfPass.SQL = "exec sp_myCoolStoreProc;"
And the stored procedure can even be passed a parameter like this:
qdfPass.SQL = "exec sp_myCoolStoreProc " & strMyParam
and then :
CurrentDb.Execute "INSERT INTO LocalTable SELECT * FROM MyPass”
So we can even use a select into/append from a store procedure by doing the above and the table/sql server side is dynamic or can even be a stored procedure. Again VERY little code.
I would suggest you avoid the idea proposed here to write recordset looping code unless one really has the desire to write looping code when none is required. And things like PK would have to be dealt with separate in code if you use such loops since the local pk column may need to be skipped (you simply leave that column out of the select SQL).
Note again that the connection string saved for the pass-though query does NOT require the user ID and password by using the above link showing how to “logon” to SQL Server. And if the table is known, then again a saved table link or pass-though query will suffice here.
I am a relative newbie and trying to insert multiple rows (and data from textboxes) from one table into another and am stuck.
This SQL identifies the data to be inserted into the table
strsql = "SELECT '" & textbox1.text & "', '" & Textbox2.text & "', "
strsql = strsql & " a.TaskNum, a.StartDay, a.NumofDays FROM VETTimeLines as a"
strsql = strsql & " ORDER BY a.StartDay"
I started out along the lines of -> Insert into StudentProgram Values() code shown above, after 3 days of trying I now look forward to your advice.
Many thanks in anticipation
Peter
Inserting data using C# can be performed in variety of ways such as using Entity Framework or using ADO.NET, as you have chosen to do in this case.
Using ADO.NET you either write the insert as you did or by using DataAdapter approach. DataAdapter is capable of creating the SQL code for you, amongst other things. For example see:SQL DataAdapter. It is a good idea to not build a SQL string as you did because of sql injection threat as #Joel Coehoorn indicates in his comment above.
One way to overcome this is by using Parameters as shown in the above link. If you decide to provide the SQL for insert yourself with parameters, here is a good example:StackOverFlow-Insert using ADO.
The idea behind all of the above code is to create a connection object, create a parameter object, create a command object, open the database connection, execute the command and close the connection.
Try one of the above approaches and let's know your specific issue if any arises.