I am having an issue where SqlBulkCopy is not copying all of the data in a column over to the destination table. I have verified that the source data (which is a .CSV file) has values in the column in all of the rows, but only the first 40 or so rows in that column are getting copied over.
The destination table's columns are set to NVARCHAR(255) and all of them are allowed to be nullable.
Here is my function to do the bulk copy:
Private Sub loadDataFromCSV(ByVal pathToFile As String, ByVal connString As String, ByVal file As String, ByVal colCount As Integer)
Dim fileLocation As String = "Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & pathToFile & ";Extended Properties='text;HDR=NO;FMT=Delimited(,)';"
Dim qry As String = "select * from " & file
Dim CompData As OleDbDataReader
Using destConnection As SqlConnection = _
New SqlConnection(connString)
destConnection.Open()
Using sourceConnection As New OleDbConnection(fileLocation)
Dim cmdSourceData As New OleDbCommand(qry, sourceConnection)
sourceConnection.Open()
CompData = cmdSourceData.ExecuteReader()
Using bulkCopy As SqlClient.SqlBulkCopy = New SqlClient.SqlBulkCopy(connString)
bulkCopy.DestinationTableName = "dbo.Records"
bulkCopy.BatchSize = 10000
bulkCopy.BulkCopyTimeout = 90
Try
bulkCopy.WriteToServer(CompData)
Catch ex As Exception
Console.WriteLine(ex.Message)
Finally
CompData.Close()
End Try
End Using
End Using
End Using
End Sub
As far as I can tell, all of the data from the table is making it over into the correct columns, with the exception of the 7th column. In the 7th column, I get the first 40 or so rows of data, and then the rest of the values for the column are NULL.
I've run out of ideas for what could be going wrong, so any help would be greatly appreciated.
Thanks.
My guess would be it is coversion errors. OLEDB will infer the datatype of a column based on the first 8 rows (I think) so if your first rows are:
SomeColumn
----------
1
2
3
4
5
6
7
8
9
apple
This initially looks like an integer column, so this is what it is mapped to, but then when it gets to "apple" it can't convert it to an integer, so returns DbNull.
The solution to this is to add IMEX=1 to your connection string, this means that no implicit conversion will be done, and the OleDbReader will just read exactly what is in the csv.
The downside of this is that you will probably then get conversion errors when trying to call the SqlBulkCopy.WriteToServer(DataReader) method. You may need to create a DataTable in the same format as your database table, and iterate over your OleDbReader doing explicit conversions where necessary, then write this DataTable to the database with SqlBulkCopy.
As a hacky workaround I simply put a character string into each column in the top 8 rows of my csv file. It fools sqlbulkcopy into treating all fields as strings. Then in sql delete the records that contain the character string.
Related
I am working on a visual basic project. I have a mdb database connected to my project. I want to add a SELECT query that finds the results which are in array that i give it on my program
I have tried to write a statement like that:
SELECT kodu, adi_soyadi, sectigi_ders_say
FROM ogrenciler
WHERE kodu IN ?
But it does not work. In my page codes I have an array and I want to find results from "ogrenciler" table where the "kodu" is in my array.
Well, you could send that array to a temp table in Access, but that would prevent more then one user using the software at the same time. (or you could add some user name to the temp table. However, if the array of choices is small, say about max 50, then you can create the sql string.
eg:
Dim MySQL As String = "SELECT * from tblHotels WHERE ID IN("
Dim IdList(5) As Integer
Dim i As Integer
For i = 1 To 5
IdList(i) = i
Next
Dim MyList As String = ""
For i = 1 To 5
If MyList <> "" Then MyList = MyList & ","
MyList = MyList & IdList(i)
Next
MySQL = MySQL & MyList & ")"
Using MyCon2 As New OleDbConnection(My.Settings.OLESQL)
Dim da As New OleDbDataAdapter(MySQL, MyCon2)
Dim rstDat As New DataTable()
da.Fill(rstDat)
For i = 0 To rstDat.Rows.Count - 1
Debug.Print(rstDat.Rows(i).Item("HotelName"))
Next ' etc etc. etc.
End Using
So you can use the SQL format of:
SELECT * FROM tblHotels where ID IN (1,2,3)
And thus build up the "list". The only downside to this approach is that the sql string is limited to 2000 characters. So, if your list is larger then say 50 or so items, then you have to adopt a different approach.
i have a table that captures the details of Products. I want to be able to generate an Auto ID that contains special characters
I have been able to write code that generates the auto ID and also add a new one. But the issue arises when it gets to the 10th and 11th record.
It seems to be seeing the 9th record as the MAX in the database.
this makes it to throw error that there will be duplicate.
For example, record 9 generates CAT009, record 10 generates CAT0010 but instead to generate record 11 as CAT0011, it seems to keep generating CAT0010 because of the MAX function i used in the SELECT statement.
Sub auto()
Try
Dim cn As New OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=" & Application.StartupPath & "\inventory1.accdb")
cn.Open()
Dim dr As OleDbDataReader
Dim cmddr As New OleDbCommand
cmddr.CommandText = "SELECT MAX(category_id) as max_id FROM tblcategory "
cmddr.Connection = cn
dr = cmddr.ExecuteReader
dr.Read()
If IsDBNull(dr("max_id")) Then
autonumber = "CAT00" & 1
Else
Dim str As String
str = dr.Item("max_id").ToString()
Dim P As Double
Dim N As Double
N = Len(str)
P = N - 5
autonumber = "CAT00" & Convert.ToInt32(str.Substring(5, P))+ 1
End If
cn.Close()
Catch Ex As Exception
MsgBox(Ex.Message)
Console.WriteLine(Ex.Message)
End Try
End Sub
The overall concept is a duplicate of other questions, the key concern being proper formatting of the number with correct zero padding. There are various ways to do this in VB.Net, but the Int32.ToString(String) accepts a format string that can handle zero-padding. It's also unnecessary to calculate the string length, since Substring() has an overload to get all remaining characters to the right.
If IsDBNull(dr("max_id")) Then
autonumber = "CAT001"
Else
Dim str As String
str = dr.Item("max_id").ToString()
autonumber = "CAT" & (Convert.ToInt32(str.Substring(3)) + 1).ToString("000");
End If
If the first two zeros should be static (i.e. always "CAT00"), then it is critical to still format the changing portion with proper padding. Otherwise the text strings will not sort properly and so the numeric order will be ineffective. In that case, you should use something like
If IsDBNull(dr("max_id")) Then
autonumber = "CAT00001" '* NOT CAT001
Else
Dim str As String
str = dr.Item("max_id").ToString()
autonumber = "CAT00" & (Convert.ToInt32(str.Substring(5)) + 1).ToString("000");
End If
Of course in either case, be aware that the number of integer digits limits the range of valid numbers. For example, 3 digits ("000") has a maximum of 999.
A better approach is to use pure integer AutoNumbers in the database: 132 -> "CAT132". Then just format the number for display purposes, and when necessary parse user input text to extract the integer portion: "CAT0089" -> 89. This approach highly simplifies the backend code and leaves the AutoNumber functionality to the database itself... no need to have custom code to generate the numbers.
I'm trying to run a list of SQL queries where a condition exists for "code" and the values sit in a range of cells on another sheet (from cells A2 to A385).
I have the code below, however, I get an invalid object name for SQLQueries!$A2:A385
So, I understand the syntax is not correct but I'm struggling to find the correct one regardless of reading numerous articles.
Sub RunSQLQueries()
'Select SQLQueries sheet
Sheets("SQLQueries").Activate
'Initializes variables
Dim cnn As New ADODB.Connection
Dim rst As New ADODB.Recordset
Dim ConnectionString As String
Dim StrQuery As String
'Setup the connection string for accessing MS SQL database
ConnectionString = "Provider=SQLOLEDB; Data Source=HOSTNAME; Initial Catalog=DBNAME; UID=domain\user; Integrated Security=SSPI"
'Opens connection to the database
cnn.Open ConnectionString
'Timeout
cnn.CommandTimeout = 900
'Build SQK queries
StrQuery = "SELECT * FROM table WHERE code IN (SELECT * FROM [SQLQueries!$A2:A385])"
'Performs the queries
rst.Open StrQuery, cnn
'Select Results sheet
Sheets("Results").Activate
'Dumps all the results from the StrQuery into cell A2 of the active sheet
Range("A2").CopyFromRecordset rst
End Sub
The result I'm expecting is for a SQL query to be run using each condition from the range of values with the results being populated on the "Results" sheet from cells A2 down
The query string is literally sent to the database server, and since your sql attempts to refer to an excel list that the server cannot access it returns an error. The server is looking for a table named [SQLQueries!$A2:A385]
To stick with your current plan, you will need to pass the the IN () clause literally or by vba variable that is formatted as such:
IN ( 'item1', 'item2' ...)
Note:You can remove single quotes if the items are numeric
I advise rethinking the plan by either
1) if it is possible to adjust things in the database side: Can you create a new reference table to join to the actual table or create a view that only returns desired rows? Then you would need a job that tweaks the filtering view/ table before running the query. The idea being you would not need to adjust the query each time bc a constant sql string would return the rows you need
Or
2) if the source table has say 100k rows or less, and data is not too wide, just select all the rows into excel in a new sheet, then filter that sheet (add a new column in excel that returns true using vlookup against your reference sheet) or use vlookup on your reference sheet to pull the desired columns
Here's a suggestion:
StrQuery = "SELECT * FROM table WHERE code IN (" & _
InList(Sheets("SQLQueries").Range("A2:A385"),True) & ")"
Function to create a SQL "in" list given a range:
Function InList(rng As Range, quoted As Boolean)
Dim qt, a, r As Long, c As Long, rv As String, v, sep As String
a = rng.Value
qt = IIf(quoted, "'", "")
For r = 1 To UBound(a, 1)
For c = 1 To UBound(a, 2)
v = Trim(a(r, c))
If Len(v) > 0 Then
rv = rv & sep & qt & v & qt
sep = ","
End If
Next c
Next r
InList = rv
End Function
Notes:
Pass False as the second argument if you have numeric values
I wouldn't use this for very large lists
Be very certain you're not at risk from possible SQL injection issues: parameterized queries are always preferable but do not work with "in" lists
I am writing a program for my lab. Based on a salesordernumer (SO-nr) I need to find the corresponding part number out of a table in access and put it into a variable( Prob a string?). later I need to split the different parts from the partnumber but before that I need to get it out of the MS table. This is de code I use now but I get an error.
Private Sub BtnOphaal_Click(sender As Object, e As EventArgs) Handles BtnOphaal.Click
If conn.State = ConnectionState.Open Then
Dim Sonr As String
Sonr = "SELECT *FROM prodvolg "
Dim SQL As New OleDb.OleDbCommand(Sonr, conn)
Dim DataAdapter As New OleDb.OleDbDataAdapter(SQL)
Dim datatabel As New DataTable("prodvolg")
DataAdapter.Fill(datatabel)
Dim queryString As String = "SELECT [pPart] AS Partnummer FROM [prodvolg] WHERE ([pSonr]='" & txtSOnummer.Text & "')"
Dim command As New OleDbCommand(queryString, conn)
Dim reader As OleDbDataReader = command.ExecuteReader()
While reader.Read()
txtppart.Text = (reader.GetString(1))
End While
reader.Close()
End If
End Sub
As you can see I'm just a beginning programmer.
The error is occuring at txtppart.Text = (reader.GetString(1)) the error message:
A first chance exception of type 'System.IndexOutOfRangeException' occurred in System.Data.dll
Additional information: Index was outside the bounds of the array.
at school I learned programming at ADO system but the future seems to be oledb but I don't understand the OLEdb system good so far. If anyone could help me I would be so happy.
The partnumber is could look like this: "CA-017630-6.35M-1/0-2"
Arrays start at index zero, not one. You have only one field returned by the query
txtppart.Text = (reader.GetString(0))
Also, keep in mind that GetString could fail if your row at index zero contains a null value.
If this is the case I suggest to check for null values with
txtppart.Text = IF(reader.IsDBNull(0), string.Empty, reader.GetString(0))
You are retrieving just one field pPart with your SQL statement, so you'll have just one field on the DataReader. Try with reader.GetString(0), because the arrays starts at index zero
I have a RichTextBox which has only text. How can I get the data (data is tab delimited) into my sql server table ? (I am not forced to use RTB so any other options would do fine as long as the entire process will be quick)
Example of text in RTB:
John 1985 (tab) 01 (tab) 19 (tab) 1.80 (tab) 70
Tony 1988 (tab) 02 (tab) 27 (tab) 1.67 (tab) 55
Table in sql Server called "Users":
Name, Year, Month, Date, Height, Weight
I found this code Dim lines As String() = myRichTextbox.Text.Split(New Char() {"\n"c})
so i just need to get each part of every line into a specific column...
any ideas ? I can't seem to get the data from RTB into SQL SERVER table. And i can't find it online... i will loop this process every 10 minutes so i don't want to save the RTB text into a file and then read from that file to save the data in table...
Once you have parsed the data you can use dynamic SQL or a stored procedure to insert the data into the DB.
SQL server manager can build the INSERT syntax for your table e.g.:
INSERT INTO [OIS].[dbo].[Burn_Type]
([Burn_Type]
,[Record_Create_Date]
,[Last_Update_Date]
,[Burn_Fee])
VALUES
(<Burn_Type, varchar(40),>
,<Record_Create_Date, datetime,>
,<Last_Update_Date, datetime,>
,<Burn_Fee, float,>)
You would need to replace the VALUES with your data then execute the SQL, here is some general code:
Public Function UpdateBySQL(ByVal SQL As String, Optional ByVal UserConnectString As String = "") As Integer
Dim sConStr As String = UserConnectString
If sConStr.Length = 0 Then sConStr = g.OISConnectString
Dim cmd As New Data.SqlClient.SqlCommand ' text commands only
Dim iCnt As Integer
Try
' wouldn't work for transactions
Using con As New SqlConnection(sConStr)
con.Open()
cmd.CommandText = SQL
cmd.Connection = con
iCnt = cmd.ExecuteNonQuery()
con.Close()
End Using
Catch ex As Exception
MsgBox(ex.Message & vbCrLf & vbCrLf & SQL)
End Try
Return iCnt
End Function
For higher security and performance use stored procedures.
You can split on the new line character (chr(10)) to get the rows, and then on each row you can split on tab character (chr(9)) to get the "columns"
Dim rows As String() = source.Split(Chr(10))
Dim columns As String()
For Each line As String In rows
columns = line.Split(Chr(9))
Next