I need to import a large CSV file into an SQL server. I'm using this :
BULK
INSERT CSVTest
FROM 'c:\csvfile.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
problem is all my fields are surrounded by quotes (" ") so a row actually looks like :
"1","","2","","sometimes with comma , inside", ""
Can I somehow bulk import them and tell SQL to use the quotes as field delimiters?
Edit: The problem with using '","' as delimiter, as in the examples suggested is that :
What most examples do, is they import the data including the first " in the first column and the last " in the last, then they go ahead and strip that out. Alas my first (and last) column are datetime and will not allow a "20080902 to be imported as datetime.
From what I've been reading arround I think FORMATFILE is the way to go, but documentation (including MSDN) is terribly unhelpfull.
Try FIELDTERMINATOR='","'
Here is a great link to help with the first and last quote...look how he used the substring the SP
http://www.sqlteam.com/article/using-bulk-insert-to-load-a-text-file
Another hack which I sometimes use, is to open the CSV in Excel, then write your sql statement into a cell at the end of each row.
For example:
=concatenate("insert into myTable (columnA,columnB) values ('",a1,"','",b1,"'")")
A fill-down can populate this into every row for you. Then just copy and paste the output into a new query window.
It's old-school, but if you only need to do imports once in a while it saves you messing around with reading all the obscure documentation on the 'proper' way to do it.
Try OpenRowSet. This can be used to import Excel stuff. Excel can open CSV files, so you only need to figure out the correct [ConnectionString][2].
[2]: Driver={Microsoft Text Driver (*.txt; *.csv)};Dbq=c:\txtFilesFolder\;Extensions=asc,csv,tab,txt;
I know this isn't a real solution but I use a dummy table for the import with nvarchar set for everything. Then I do an insert which strips out the " characters and does the conversions. It isn't pretty but it does the job.
Id say use FileHelpers its an open source library
Do you need to do this programmatically, or is it a one-time shot?
Using the Enterprise Manager, right-click Import Data lets you select your delimiter.
You have to watch out with BCP/BULK INSERT because neither BSP or Bulk Insert handle this well if the quoting is not consistent, even with format files (even XML format files don't offer the option) and dummy ["] characters at the beginning and end and using [","] as the separator. Technically CSV files do not need to have ["] characters if there are no embedded [,] characters
It is for this reason that comma-delimited files are sometimes referred to as comedy-limited files.
OpenRowSet will require Excel on the server and could be problematic in 64-bit environments - I know it's problematic using Excel in Jet in 64-bit.
SSIS is really your best bet if the file is likely to vary from your expectations in the future.
u can try this code which is very sweet if you want ,
this will remove unwanted semicolons from your code.
if for example your data is like this :"Kelly","Reynold","kelly#reynold.com"
Bulk insert test1
from 'c:\1.txt' with (
fieldterminator ='","'
,rowterminator='\n')
update test1<br>
set name =Substring (name , 2,len(name))
where name like **' "% '**
update test1
set email=substring(email, 1,len(email)-1)
where email like **' %" '**
Firs you need to import CSV file into Data Table
Then you can insert bulk rows using SQLBulkCopy
using System;
using System.Data;
using System.Data.SqlClient;
namespace SqlBulkInsertExample
{
class Program
{
static void Main(string[] args)
{
DataTable prodSalesData = new DataTable("ProductSalesData");
// Create Column 1: SaleDate
DataColumn dateColumn = new DataColumn();
dateColumn.DataType = Type.GetType("System.DateTime");
dateColumn.ColumnName = "SaleDate";
// Create Column 2: ProductName
DataColumn productNameColumn = new DataColumn();
productNameColumn.ColumnName = "ProductName";
// Create Column 3: TotalSales
DataColumn totalSalesColumn = new DataColumn();
totalSalesColumn.DataType = Type.GetType("System.Int32");
totalSalesColumn.ColumnName = "TotalSales";
// Add the columns to the ProductSalesData DataTable
prodSalesData.Columns.Add(dateColumn);
prodSalesData.Columns.Add(productNameColumn);
prodSalesData.Columns.Add(totalSalesColumn);
// Let's populate the datatable with our stats.
// You can add as many rows as you want here!
// Create a new row
DataRow dailyProductSalesRow = prodSalesData.NewRow();
dailyProductSalesRow["SaleDate"] = DateTime.Now.Date;
dailyProductSalesRow["ProductName"] = "Nike";
dailyProductSalesRow["TotalSales"] = 10;
// Add the row to the ProductSalesData DataTable
prodSalesData.Rows.Add(dailyProductSalesRow);
// Copy the DataTable to SQL Server using SqlBulkCopy
using (SqlConnection dbConnection = new SqlConnection("Data Source=ProductHost;Initial Catalog=dbProduct;Integrated Security=SSPI;Connection Timeout=60;Min Pool Size=2;Max Pool Size=20;"))
{
dbConnection.Open();
using (SqlBulkCopy s = new SqlBulkCopy(dbConnection))
{
s.DestinationTableName = prodSalesData.TableName;
foreach (var column in prodSalesData.Columns)
s.ColumnMappings.Add(column.ToString(), column.ToString());
s.WriteToServer(prodSalesData);
}
}
}
}
}
This is an old question, so I write this to help anyone who stumble upon it.
SQL Server 2017 introduces the FIELDQUOTE parameter which is intended for this exact use case.
Yup, K Richard is right: FIELDTERMINATOR = '","'
See http://www.sqlteam.com/article/using-bulk-insert-to-load-a-text-file for more info.
You could also use DTS or SSIS.
Do you have control over the input format? | (pipes), and \t usually make for better field terminators.
If you figure out how to get the file parsed into a DataTable, I'd suggest the SqlBulkInsert class for inserting it into SQL Server.
Related
I have a SQL Server table with the following design:
I need to export this entire table to a CSV file, with the data looking like this in plain-text format:
When I use the Import/Export wizard in SSMS, I get the following results in the CSV:
As you can see, this doesn't contain the actual binary data from the Image columns.
Instead, it just says System.Byte[] in plain-text, in each row.
In the past I've copy/pasted the text directly from the SSMS results window, into notepad, to manually produce the CSV.
However this table is so long, and the images are so large, that my computer runs out of memory if I try this now.
There must be a method to export the binary data into a CSV, in 0x1234ABCD format (hex).
How could I accomplish this?
if you are a developed you can create small C# app Useing Base64 encoding for your binary data. It converts any binary data into readable string. The code is as following
Encode
public static string Base64Encode(string plainText)
{
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
return System.Convert.ToBase64String(plainTextBytes);
}
Decode
public static string Base64Decode(string base64EncodedData)
{
var base64EncodedBytes = System.Convert.FromBase64String(base64EncodedData);
return System.Text.Encoding.UTF8.GetString(base64EncodedBytes);
}
Following the advice in Dan Guzman's comment, I created a new view with the following query:
CREATE VIEW vwPhotosMen AS
SELECT SS,
CONVERT(varchar(max), CONVERT(varbinary(MAX), PhotoMen, 1), 1) AS PhotoMen,
CONVERT(varchar(max), CONVERT(varbinary(MAX), Thumbnail, 1), 1) AS Thumbnail
FROM dbo.PhotosMen
The SSMS Export Wizard was then able to export the data correctly from this view, to a plain-text CSV file.
I'm a newbie so don't laugh :#
I'm working with 2002-2003 Microsoft access database.
Now, I want to add an array of DataRow into an existing table that I've in my database. Is there a way to do that? because right now I'm just adding the rows with a foreach loop
thank you
I think that the foreach-loop actually is the best way to do it.
foreach(DataRow row in yourRowArray)
{
dataTable.Add(row);
}
If you are using .Net Framework 3.5+ you can also use the DataRows CopyToDataTable() Method.
But you have to watch out because the Data in the DataTable is overwritten in this case.
DataTable table = yourDataTable;
DataRow[] yourRowArray = ...;
if(yourRowArray.Length > 0)
{
table = yourRowArray.CopyToDataTable();
}
I would recommend using the foreach-loop.
What you describe as array must be a saved file type i.e. excel or csv. Be sure it is a clean grid of data without extraneous non aligned rows.
Then you can link to that file with Access as a table. This is a manual step using the Access interface - in the ribbon it is the External area. This link remains good - allowing you to replace the excel/csv with a new one as long as the location path and structure of the file do not change.
Then you create an Append query to write all the records from this table into the table in your Access database.
www.CahabaData.com
I use a text file with a correct sql insert-statement for bulk loading in my test project. I apply this statement to an empty table to load data. But I faced with the problem of writing an array of bytes.
How can I write the correct insert-statement that I used to pass an array of bytes? Will understand SQL Server 2008 an array of bytes as hex-string?
Added
I need some string that I can apply to SQL Server to insert an array of bytes.
Something like that:
INSERT INTO T(SomeBlobColumn) VALUES (0x666F6F)
or
INSERT INTO T(SomeBlobColumn) VALUES (CAST('666F6F' AS BINARY))
You could try this approach (in c#):
using(SqlCommand cmd = new SqlCommand("INSERT INTO TableName(BinaryColumn) VALUES (#InputParameter)", sqlConnection))
{
cmd.Parameters.Add("#InputParameter", SqlDbType.VarBinary, 8000).Value = byteArray;
cmd.ExecuteNonQuery();
}
I'm working with an Oracle database, and I would like to read the contents of a BLOB. How do I do this?
When I do a simple select statement, it merely returns "(BLOB)" (without the quotes). How do I read the actual contents?
You can dump the value in hex using UTL_RAW.CAST_TO_RAW(UTL_RAW.CAST_TO_VARCHAR2()).
SELECT b FROM foo;
-- (BLOB)
SELECT UTL_RAW.CAST_TO_RAW(UTL_RAW.CAST_TO_VARCHAR2(b))
FROM foo;
-- 1F8B080087CDC1520003F348CDC9C9D75128CF2FCA49D1E30200D7BBCDFC0E000000
This is handy because you this is the same format used for inserting into BLOB columns:
CREATE GLOBAL TEMPORARY TABLE foo (
b BLOB);
INSERT INTO foo VALUES ('1f8b080087cdc1520003f348cdc9c9d75128cf2fca49d1e30200d7bbcdfc0e000000');
DESC foo;
-- Name Null Type
-- ---- ---- ----
-- B BLOB
However, at a certain point (2000 bytes?) the corresponding hex string exceeds Oracle’s maximum string length. If you need to handle that case, you’ll have to combine How do I get textual contents from BLOB in Oracle SQL with the documentation for DMBS_LOB.SUBSTR for a more complicated approach that will allow you to see substrings of the BLOB.
SQL Developer can show the blob as an image (at least it works for jpegs). In the Data view, double click on the BLOB field to get the "pencil" icon. Click on the pencil to get a dialog that will allow you to select a "View As Image" checkbox.
If the content is not too large, you can also use
SELECT CAST ( <blobfield> AS RAW( <maxFieldLength> ) ) FROM <table>;
or
SELECT DUMP ( CAST ( <blobfield> AS RAW( <maxFieldLength> ) ) ) FROM <table>;
This will show you the HEX values.
If you use the Oracle native data provider rather than the Microsoft driver then you can get at all field types
Dim cn As New Oracle.DataAccess.Client.OracleConnection
Dim cm As New Oracle.DataAccess.Client.OracleCommand
Dim dr As Oracle.DataAccess.Client.OracleDataReader
The connection string does not require a Provider value so you would use something like:
"Data Source=myOracle;UserID=Me;Password=secret"
Open the connection:
cn.ConnectionString = "Data Source=myOracle;UserID=Me;Password=secret"
cn.Open()
Attach the command and set the Sql statement
cm.Connection = cn
cm.CommandText = strCommand
Set the Fetch size. I use 4000 because it's as big as a varchar can be
cm.InitialLONGFetchSize = 4000
Start the reader and loop through the records/columns
dr = cm.ExecuteReader
Do while dr.read()
strMyLongString = dr(i)
Loop
You can be more specific with the read, eg dr.GetOracleString(i) dr.GetOracleClob(i) etc. if you first identify the data type in the column. If you're reading a LONG datatype then the simple dr(i) or dr.GetOracleString(i) works fine. The key is to ensure that the InitialLONGFetchSize is big enough for the datatype. Note also that the native driver does not support CommandBehavior.SequentialAccess for the data reader but you don't need it and also, the LONG field does not even have to be the last field in the select statement.
What client do you use? .Net, Java, Ruby, SQLPLUS, SQL DEVELOPER? Where did you write that simple select statement?
And why do you want to read the content of the blob, a blob contains binary data so that data is unreadable. You should use a clob instead of a blob if you want to store text instead of binary content.
I suggest that you download SQL DEVELOPER: http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html . With SQL DEVELOPER you can see the content.
If you're interested to get the plaintext (body part) from a BLOB, you could use the CTX_DOC package.
For example, the CTX_DOC.FILTER procedure can "generate either a plain text or a HTML version of a document". Be aware that CTX_DOC.FILTER requires an index on the BLOB column. If you don't want that, you could use the CTX_DOC.POLICY_FILTER procedure instead, which doesn't require an index.
I basically need to know how to import SQL code into Access. I've tried one way but that requires me to do one table and one value at a time which takes a lot of time.
Can anyone help?
If you are trying to import data, rather than SQL code (see Duffymo's response), there are two ways.
One is to go where the data is and dump a .CSV file and import that, as Duffymo responded.
The other is to create a table link from the Access database to a table in the source database. If the two databases will talk to each other this way, you can use the data in the remote table as if it were in the Access database.
Well, some days ago I needed to shift data from an Access database to SQL (reverse of what you're doing). I found it simpler to write a simple script that would read data from my access database and insert it into SQL.
I don't think doing what you need to do is any different.
I don't know if it will help, but I posting my code (It's a simple C# function). You can just change the connections and it will work. Of course I only had 3 fields so I hard-coded them. You can do the same for your db schema.
protected void btnProcess_Click(object sender, EventArgs e)
{
//Open the connections to the access and SQL databases
string sqlDBCnn = #"Data Source=.\SQLEXPRESS;Integrated Security=True;AttachDBFileName=|DataDirectory|\mydb.mdf;user instance=true";
string accessDBCnn = #"Provider=Microsoft.Jet.OleDB.4.0;Data Source=C:\mydb.mdb";
OleDbConnection cnnAcc = new OleDbConnection(accessDBCnn);
cnnAcc.Open();
SqlConnection cnnSql = new SqlConnection(sqlDBCnn);
cnnSql.Open();
SqlCommand cmSql = new SqlCommand("DELETE tablename", cnnSql);
cmSql.ExecuteNonQuery();
//Retrieve the data from the Access Database
OleDbCommand cmdAcc = new OleDbCommand("SELECT * FROM tablename", cnnAcc);
OleDbDataReader drAcc = cmdAcc.ExecuteReader();
using (drAcc)
{
if (drAcc.HasRows)
{
//Loop through the access database records and add them to the database
while (drAcc.Read())
{
SqlCommand cmdSql = new SqlCommand("INSERT INTO tablename(Category, Head, Val) VALUES(#cat,#head,#val)",cnnSql);
SqlParameter parCat = new SqlParameter("cat",System.Data.SqlDbType.VarChar,150);
SqlParameter parHead = new SqlParameter("head",System.Data.SqlDbType.VarChar,150);
SqlParameter parVal = new SqlParameter("val",System.Data.SqlDbType.VarChar);
parCat.Value = drAcc["Category"].ToString();
parHead.Value = drAcc["Head"].ToString();
parVal.Value = drAcc["Val"].ToString();
cmdSql.Parameters.Add(parCat);
cmdSql.Parameters.Add(parHead);
cmdSql.Parameters.Add(parVal);
cmdSql.ExecuteNonQuery();
}
}
}
lblMsg.Text = "<p /> All Done Kapitone!";
}
SQL code? Or data? "one table and one value" makes me think it's the latter. If so, I'd suggest dumping the data out into a .csv file and importing that into Access tables.
Or maybe using a tool like Microsoft's DTS to map and move the data between sources. That would be the best idea.
I guess you are talking about "importing" both structure and data from SQL to ACCESS. ACCESS does not accept standard TSQL scripts that you could generate directly from your SQL Database. There are some commercial products like EMS that can more or less do the job for you. EMS has a data exporter module that can take your SQL data in different formats, including Access.
Another way would be to open an Access file and write some basic VBA code, taking advantage of the DoCmd.TransferDatabase method, where you can link OR copy tables from other databases into Access.
I forgot if these methods also allow the transfer of a 'clean' database model, including primary keys and relations... You'll have to give it a try.