I have a large csv file with lots of data that I need to be able to analysis (~6M rows). I want to connect to the file and run SQL command against it to return only the data I'm interested in analysing. The VBA I'm writing is in Excel 2010.
Everything works fine when the number of rows in the csv file is < 4432669. When the csv file has more rows than this, the command seem to terminate at that point in the file and just returns what ever it has found up to that point. No Error is thrown (CN.Errors), I first though it might be that the command timedout but when I increase this it made no difference. I also checked with different csv files just incase that row contained corrupted data, but no luck. Recordset maxrecords is set to 0 (No limit).
I've tried using Microsoft.Jet.OLEDB.4.0; and driver={Microsoft Text Driver (*.txt; *.csv)}; in the connectionstring, both behave the same as described above.
Here is test code I'm using,
Dim CN As New ADODB.Connection
Dim RS As New ADODB.Recordset
Dim Err As ADODB.Error
providerstr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\cygwin\home\MarkM\csvimport\filtertest4\;" & _
"Extended Properties=" & Chr(34) & "text;HDR=Yes;FMT=Delimited" & Chr(34) & ";"
CN.ConnectionString = providerstr
CN.Mode = adModeRead
CN.CommandTimeout = 900
CN.Open
RS.Open "SELECT exCode FROM 5M_MBP1R04.csv", CN, adOpenStatic, adLockReadOnly
RS.MoveLast
MsgBox "Number of rows = " & RS.RecordCount
For Each Err In CN.Errors
strError = "Error #" & Err.Number & vbCr & _
" " & Err.Description & vbCr & _
" (Source: " & Err.Source & ")" & vbCr & _
" (SQL State: " & Err.SqlState & ")" & vbCr & _
" (NativeError: " & Err.NativeError & ")" & vbCr
If Err.HelpFile = "" Then
strError = strError & " No Help file available"
Else
strError = strError & _
" (HelpFile: " & Err.HelpFile & ")" & vbCr & _
" (HelpContext: " & Err.HelpContext & ")" & _
vbCr & vbCr
End If
Debug.Print strError
Next
Really appreciate any help as I'm completely stuck now.
BR's Mark.
Perhaps you are exceeding a memory constraint due to the CursorType. Try changing it to adOpenForwardOnly
Here is the MSDN page describing Cursor Types.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms681771(v=vs.85).aspx
Related
I have VBA code working in Excel on my machine but it is giving the above error on someone else's machine. What would be causing it to work for me but not for someone else?
'---Connecting to the Data Source---
Set cn = CreateObject("ADODB.Connection")
With cn
.Provider = "Microsoft.ACE.OLEDB.12.0"
.ConnectionString = "Data Source=" & wb.Path & "\" & wb.Name & ";" & "Extended Properties=""Excel 12.0 Xml;HDR=YES"";"
.Open
End With
'---Run the SQL SELECT Query---
sql = "SELECT " & _
"sum([" & cash_projections.Name & "$].[Projected Ending Balance]), " & _
"[" & cash_projections.Name & "$].[Account Name (Balance)], " & _
"[" & cash_projections.Name & "$].[Projected Date] " & _
"FROM " & _
"[" & cash_projections.Name & "$] " & _
"WHERE " & _
"[" & cash_projections.Name & "$].[Projected Ending Balance] > 100 " & _
"GROUP BY " & _
"[" & cash_projections.Name & "$].[Account Name (Balance)], " & _
"[" & cash_projections.Name & "$].[Projected Date] " & _
"ORDER BY " & _
"[" & cash_projections.Name & "$].[Account Name (Balance)], " & _
"[" & cash_projections.Name & "$].[Projected Date] "
Set rs = cn.Execute(sql)
The error shows up on the last line of this code. Again, it works just fine for me, but gives the error on another machine.
Please note that I am using the exact same workbook as is being used on the other computer. Here are the column headings
The line that is causing the error is the one that is executing your query. Because the code does not fail when you open the connection, it means it is finding the workbook you are querying just fine. My guess is that the workbook it finds has a different structure on the two machines. Because it says a parameter is missing a value, it means that something that is defined on your machine is missing on the other, it could be a table name or a column name. Basically, when you use that name on your machine it finds it and moves on. On the other machine, it can't find it so it assumes it must be parameter.
In the process of building a Database for my employer.
This has been going mostly smoothly, but I have had an issue where a table "updates" but no record is inserted.
... well sort of not inserted.
I can't actually tell if it is or not. I mean it's not on the table, and my SQL query returns a 0 BUT my VBA SQL query returns a 1 on a record I told it to insert.
Code below
Dim rst As Recordset
Dim wrk As DAO.Workspace
Dim db As DAO.Database
Dim rsp As Integer
Dim uTable As String
Dim inSQL As String
Debug.Print pCurrentCmd
If pCurrentCmd = "NewIT" Then
Set rst = CurrentDb.OpenRecordset("SELECT COUNT(*) FROM tbl_IT_Staff " & _
"WHERE GivenName LIKE " & Chr(34) & Chr(42) & _
Me.txt_GivenName & Chr(42) & Chr(34) & _
" AND Surname LIKE " & Chr(34) & Chr(42) & _
Me.txt_Surname & Chr(42) & Chr(34) & ";")
If rst.Fields(0) > 0 Then
rsp = MsgBox("This person is already in the database!" & _
vbCrLf & "Do you wish to proceed anyway? ", vbYesNo)
Else
Set rst = Nothing
GoTo Finalize
End If
If rsp = 6 Then GoTo Finalize
If rsp = 7 Then GoTo exitSub
Set rst = Nothing
End If
Finalize:
Set wrk = DBEngine.Workspaces(0)
Set db = wrk.OpenDatabase(CurrentDb.Name)
DoEvents
With wrk
On Error GoTo ErrRollback
If pCurrentCmd = "NewIT" Then
uTable = "tbl_IT_Staff"
End If
.BeginTrans
If IsNull(Me.txt_AOR) Then
inSQL = "INSERT INTO " & uTable & " (GivenName, Surname) VALUES (" & _
Chr(34) & Me.txt_GivenName & Chr(34) & Chr(44) & Chr(34) & _
Me.txt_Surname & Chr(34) & ");"
Else
inSQL = "INSERT INTO " & Chr(34) & uTable & Chr(34) & _
" (GivenName, Surname) VALUES (" & Chr(34) & Me.txt_GivenName & _
Chr(34) & Chr(44) & Chr(34) & Me.txt_Surname & Chr(34) & _
Chr(44) & Chr(34) & Me.txt_AOR & Chr(34) & ");"
End If
Debug.Print inSQL
CurrentDb.Execute inSQL, dbFailOnError
Call AuditLogR(DMax("ID", "tbl_IT_Staff"), "NewP", uTable)
.CommitTrans
End With
exitSub:
Exit Sub
Now I created a new test record : Test Tests and ran the code, it went through, and committed the transaction, BUT, no record exists in the table.
Ok, nbd, I made a change (had commented out the auditlog call, so put it back in) then decided to rerun the code, and see if the audit log updates or not (this has been a constant issue in every new instance of running the audit log, it "updates" but adds no record to the table.) This is the first case where a different table is claiming to have updated, but clearly has not. Often with the Auditlog it has been easily solved.
But what's throwing me off is the fact that this record both exists, and doesn't exist. and that using the same exact SQL query, I get different results depending on if it is done via VBA (where it is 'failing?') and SQL which matches the visible records on the table.
Anyone have any suggestions? Or a direction they can point me in?
I have a series of codes that run when designated, the first is a connection check. If it is successful then it allows the code to continue, if not it stops it altogether. I am worried however about what happens when the connection is lost after this process. There is data on a local table that is uploaded to our SQL server during this process, if the connection terminates mid download it looks like sometimes the data is still transferred but not if it happens right away.
The second part of the code, deletes all the local tables contents which contain employee information, then downloads the new data so if there were any updates the most recent information is provided.
I am trying to figure out if there is a method or code that can be implemented to tell the query to stop running as soon as connection is lost, or if there is a way to undo it if it happens.
Or would it be a good idea to combine the connection code with the upload and delete codes so it runs every time before it initiates a process?
The connection code that runs at the beginning is:
Public Function StartUp()
Dim cnn As ADODB.Connection
Dim localrst As New ADODB.Recordset
Dim remoterst As New ADODB.Recordset
On Error Resume Next
Set cnn = New ADODB.Connection
cnn.Open "Provider=PRO; Data Source=SOURCE; Initial Catalog=CAT;" _
& "User Id=ID; Password=PW;"
If cnn.State = adStateOpen Then
MsgBox ("You have an established connection with the L&TD SQL Server Database and the CDData table has been uploaded to the server.")
Else
MsgBox ("Cannot connect to SQL Server. Data will be stored locally to CDData Table until application is opened again with an established connection.")
End
End If
On Error GoTo 0
' MsgBox ("Please wait while the database is updating, this may take a moment.")
End Function
As you can see, I placed an END before the END IF so if there is no connection it just ends altogether.
The UPLOAD code is
Public Function Update()
Dim cdb As DAO.Database, qdf As DAO.QueryDef
Dim rs As DAO.Recordset
Dim err As DAO.Error
' Const DestinationTableName = "AC_CDData"
Const ConnectionString = _
"ODBC;" & _
"Driver={SQL Server Native Client 10.0};" & _
"Server=SERV;" & _
"Database=DB;" & _
"UID=ID;" & _
"PWD=PWD;"
Set cdb = CurrentDb
Set qdf = cdb.CreateQueryDef("")
Set rs = CurrentDb.OpenRecordset("CDData", dbOpenTable)
qdf.Connect = ConnectionString
Do While Not rs.EOF
qdf.SQL = "INSERT INTO AC_CDData_1(EmployeeID, EmployeeName, Region, District, Function1, Gender, EEOC, Division, Center, MeetingReadinessLevel, ManagerReadinessLevel, EmployeeFeedback, DevelopmentForEmployee1, DevelopmentForEmployee2, DevelopmentForEmployee3, DevelopmentForEmployee4, DevelopmentForEmployee5, Justification, Notes, Changed, JobGroupCode, JobDesc, JobGroup) " & _
"Values (" & _
"'" & rs!EmployeeID & "', " & _
"'" & rs!EmployeeName & "', " & _
"'" & rs!Region & "', " & _
"'" & rs!District & "', " & _
"'" & rs!Function1 & "', " & _
"'" & rs!Gender & "', " & _
"'" & rs!EEOC & "', " & _
"'" & rs!Division & "', " & _
"'" & rs!Center & "', " & _
"'" & rs!ManagerReadinessLevel & "', " & _
"'" & rs!MeetingReadinessLevel & "', " & _
"'" & rs!EmployeeFeedback & "', " & _
"'" & rs!DevelopmentForEmployee1 & "', " & _
"'" & rs!DevelopmentForEmployee2 & "', " & _
"'" & rs!DevelopmentForEmployee3 & "', " & _
"'" & rs!DevelopmentForEmployee4 & "', " & _
"'" & rs!DevelopmentForEmployee5 & "', " & _
"'" & rs!Justification & "', " & _
"'" & rs!Notes & "', " & _
"'" & rs!Changed & "', " & _
"'" & rs!JobGroupCode & "', " & _
"'" & rs!JobDesc & "', " & _
"'" & rs!JobGroup & "')"
qdf.ReturnsRecords = False
On Error GoTo Update_qdfError
qdf.Execute dbFailOnError
On Error GoTo 0
rs.MoveNext
Loop
rs.Close
Set qdf = Nothing
Set cdb = Nothing
Set rs = Nothing
Exit Function
Update_qdfError:
For Each err In DAO.Errors
MsgBox err.Description, vbCritical, "Error " & err.Number
Next
End Function
So is there a way I can modify the Connection code and add it to the update code (minus the message boxes) so if the connection cuts off it will terminate the code?
Did you try out transactions? Wrapping your insert script inside a transaction will not change the database until you've explicitly committed the transaction. If the data connection is lost during the inserts, the commit will never be called, and the SQL Server data won't be changed.
See http://msdn.microsoft.com/en-us/library/office/ff196400%28v=office.15%29.aspx
I'm trying to connect to a SQL server(localhost) and extract the following:
SELECT INVOICENO, SUPPLIERID, AMOUNT, DOCID
FROM ES_TRANS_HEADER
WHERE READYTOINVOICE = 1
I then need to get this information into a .txt file, seperated by commas
This is what I got so far:
function accounting()
Dim myStream, connection, myCommand, recValue
Set myStream = CreateObject("ADODB.Stream")
Set connection = CreateObject("ADODB.Connection")
Set myCommand = CreateObject("ADODB.Command")
connection.Open "Provider=SQLNCLI10;" & _
"Data Source=localhost;" & _
"Integrated Security=SSPI;" & _
"Initial Catalog=SQLDatabase;" & _
"User ID=;Password="
myCommand.ActiveConnection=connection
myCommand.CommandText="SELECT INVOICENO,SUPPLIERID, AMOUNT, DOCID FROM ES_TRANS_HEADER WHERE READYTOINVOICE = 1"
SET recValue = myCommand.Execute()
If Not recValue.EOF then
MsgBox "INVOICENO = " & recValue(0) & vbcrlf & "SUPPLIERID=" & recValue(1) _
& "AMOUNT=" & recValue(2) & "DOCID=" & recValue(3)
End If
While Not recValue.EOF
INVOICENO = recValue(0)
SUPPLIERID = recValue(1)
AMOUNT = recValue(2)
DOCCID = recValue(3)
recValue.MoveNext
Wend
end function
Am I on the right track here, and if so what should I try to do next?
Thanks in advance
MsgBox didn't have any parentheses missing. Remove them (and read Eric Lippert's most awesome blog post on parentheses in VBScript). What's missing in that line is a line continuation character (_) at the end of the line. In VBScript you cannot wrap lines without that. Change this:
MsgBox ("INVOICENO = " & recValue(0) & vbcrlf & "SUPPLIERID=" & recValue(1)
& "AMOUNT=" & recValue(2) & "DOCID=" & recValue(3))
into this:
MsgBox "INVOICENO = " & recValue(0) & vbcrlf & "SUPPLIERID=" & recValue(1) _
& "AMOUNT=" & recValue(2) & "DOCID=" & recValue(3)
or put the whole statement in a single line:
MsgBox "INVOICENO = " & recValue(0) & vbcrlf & "SUPPLIERID=" & recValue(1) & "AMOUNT=" & recValue(2) & "DOCID=" & recValue(3)
As for the database connection, your connection string seems odd. According to the information provided here that connection string is for connections to a mirrored database. Try this instead:
db = "..." 'specify database name here
connection.Open "Provider=SQLNCLI10;" _
& "Server=localhost;" _
& "Database=" & db & ";" _
& "Trusted_Connection=yes;"
If that doesn't help: check the value of the connection's State property:
WScript.Echo connection.State
Also, update your question with any error you're getting (error number, error message and line number).
In MS Access QBE if I paste the following SQL, it works correctly and I get 2 records back-
SELECT [tmp_binning].[bn_faibash] FROM [tmp_binning] WHERE key2='0210043-HOU-STOR' ORDER BY [tmp_binning].[bn_faibash];
But if I programmatically run the same query in VBA from an ADO object I get (incorrectly) no records. If I change the SQL to remove brackets around the field name, it does correctly return the 2 records in VBA ADO.
SELECT [tmp_binning].bn_faibash FROM [tmp_binning] WHERE key2='0210043-HOU-STOR' ORDER BY [tmp_binning].bn_faibash;
I've been unsuccessful googling to figure why this happens on my own, can anyone tell me why?
Thanks.
First, the brackets aren't required, either in the in Access UI or via ADO. Simply omit them in all environments and the problem should go away. (If it is the Access QBE thing that is adding the brackets then consider another tool or hand crafting your SQL code!)
Second, even with the brackets I can't reproduce the error using your SQL code e.g.
Sub gjskdjs()
On Error Resume Next
Kill Environ$("temp") & "\DropMe.mdb"
On Error GoTo 0
Dim cat
Set cat = CreateObject("ADOX.Catalog")
With cat
.Create _
"Provider=Microsoft.Jet.OLEDB.4.0;" & _
"Data Source=" & _
Environ$("temp") & "\DropMe.mdb"
With .ActiveConnection
Dim Sql As String
Sql = _
"CREATE TABLE tmp_binning" & vbCr & "(" & vbCr & " bn_faibash VARCHAR(255)," & _
" " & vbCr & " key2 VARCHAR(255)" & vbCr & ");"
.Execute Sql
Sql = _
"INSERT INTO tmp_binning (bn_faibash, key2)" & _
" VALUES ('002', '0210043-HOU-STOR');"
.Execute Sql
Sql = _
"INSERT INTO tmp_binning (bn_faibash, key2)" & _
" VALUES ('001', '0210043-HOU-STOR');"
.Execute Sql
Sql = _
"SELECT [tmp_binning].bn_faibash " & vbCr & " FROM" & _
" [tmp_binning] " & vbCr & " WHERE key2 = '0210043-HOU-STOR'" & _
" " & vbCr & " ORDER " & vbCr & " BY [tmp_binning].bn_faibash;"
Dim rs
Set rs = .Execute(Sql)
MsgBox rs.GetString
End With
Set .ActiveConnection = Nothing
End With
End Sub
Consider posting your schema as SQL DDL with sample data.