I want to import several CSV files automatically using SQL-code (i.e. without using the GUI). Normally, I know the dimensions of my CSV file. So, in many cases I create an empty table with, let say, x columns with the corresponding data types. Then, I import the CSV file into this table using BULK INSERT. However, in this case I don't know much about my files, i.e. information about data types and dimensions are not given.
To summerize the problem:
I receive a file path, e.g. C:...\DATA.csv. Then, I want to use this path in SQL-code to import the file to a table without knowing anything about it.
Any ideas on how to solve this problem?
Use something like this:
BULK INSERT tbl
FROM 'csv_full_path'
WITH
(
FIRSTROW = 2, --Second row if header row in file
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'error_file_path',
TABLOCK
)
If columns are not known, you could try with:
select * from OpenRowset
Or, do a bulk insert with only the first row as one big column, then parse it to create the dynamic main insert. Or bulk insert the whole file into a table with just one column, then parse that...
You can use OPENROWSET (documantation).
SELECT *
INTO dbo.MyTable
FROM
OPENROWSET(
BULK 'C:\...\mycsvfile.csv',
SINGLE_CLOB) AS DATA;
In addition, you can use dynamic SQL to parameterize table name and location of csv file.
The resultant error is:
Msg 207, Level 16, State 1, Line 9
Invalid column name 'Email'.
Code:
-- Bulk insert data from csv file into server temp table
BULK INSERT vwTemporaryIT_USE_ONLY_Import FROM 'C:\Bulk\b_email.csv'
WITH (
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
Go
-- Set the flag in db for all records imported from csv
UPDATE [APTIFY].[dbo].[Person]
SET
[IT_Use_Only] = 1
WHERE
[Email] IN
(Select [Email] From vwTemporaryIT_USE_ONLY_Import)
Go
I can see that the vwTemporaryIT_USE_ONLY_Import table is being populated with the data from the CSV fine, but is seems the following statement is failing for some reason:
WHERE
[Email] IN
(Select [Email] From vwTemporaryIT_USE_ONLY_Import)
I am certainly not an expert at this and I may not have setup the table or view correctly, as I recently added the Email column to both. But they have matching datatype of nvchar(100) not null. I have also tried it as null. I'm not even sure if IN handles nvchar such is the level of my SQL expertise. Any clues what I'm doing wrong?
Actually no! In the Person table it was called Email1. I have changed the code to:
WHERE [Email1]
IN
(Select [Email] From wTemporaryIT_USE_ONLY_Import)
...and now it works fine. Thanks for your help!!
Ive been working with this for a while and can't find out what I'm doing wrong. I have a CSV file with data such as
123,Jon,Son,M,1
When I run the query
BULK INSERT MYDB2..Dependent FROM 'c:\db3\db.csv'
WITH
(FIELDTERMINATOR=',',
ROWTERMINATOR = '/n')
I get errors like
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 5 (AGE).
The thing is I made EXACT copies of the tables so there is no way my tables can't match.
I believe the problem is my the format of my query.
It does have a little problem, it should be \n, instead of /n
BULK INSERT [Dependent] FROM 'c:\db3\db.csv'
WITH
(FIELDTERMINATOR=',' ,ROWTERMINATOR = '\n')
This is my source data in CSV format:
4,23,2AY5623,7235623
4,23,2GP1207,1451207
4,23,2GQ6689,4186689
Table:
CREATE TABLE [dbo].[Table1](
[idCodeLevel] [int] NOT NULL,
[idFirm] [int] NOT NULL,
[valCodeFrom] [varchar](15) NOT NULL,
[valCodeTo] [varchar](15) NOT NULL
) ON [PRIMARY]
This the code I am using to bulk import:
USE Test
GO
TRUNCATE TABLE Table1
GO
BULK INSERT Table1
FROM 'C:\Temp\test.csv'
WITH (
FIELDTERMINATOR = ',',
MAXERRORS=0,
ROWTERMINATOR = '\n'
)
GO
Error I am getting is:
Msg 4864, Level 16, State 1, Line 2
Bulk load data conversion error (type mismatch or invalid character for the specified codepage) for row 1, column 1 (idCodeLevel).
Can you please someone tell me why is it failing?
I googled and found out that I might have to use the format .fmt. But how can I convert a csv file to fmt. I have seen code to create fmt file from sql table.
Thanks a lot for your help!
Does the csv have a row at the top of field names? If so you'll need to add "FIRSTROW = 2" to your bulk statement. If not, try creating a new table that is all VARCHAR fields, then check the data: you probably have something strange in your data that you aren't expecting, like a non-printing character. Import as text and then try something like "SELECT ISNUMERIC([FIELD1]) FROM NEWTABLE".
use the sql import wizard to import data from external file.
Right click on database--->task--->import----> specify the flat file as source and select the destination server.
for more information please visit Import CSV data to SQL
I have a CSV file that contains stock quotes. I am new when it comes to SQL, but I have done a lot of research and come up with a code that I thought should work. But it doesn't. I get errors all the way....
USE ShakeOut
GO
CREATE TABLE CSVTest1
(Ticker varchar(10),
dateval smalldatetime),
timevale time(),
Openval varchar(10),
Highval varchar(10),
Lowval varchar(10),
Closeval varchar(10),
Volume varchar(10),
)
GO
BULK
INSERT CSVTest1
FROM 'c:\TEST.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
--Check the content of the table.
SELECT *
FROM CSVTest1
GO
--Drop the table to clean up database.
DROP TABLE CSVTest1
GO
My CSV file has timevalue as 03:15:00 PM, and I'm not sure how to set that up in the table. The other values I think are aproxmately right, here's a sample of my csv file:
5/1/2009,9:30:00 AM,18.21,18.45,18.21,18.32,32163
5/1/2009,9:35:00 AM,18.33,18.34,18.27,18.29,36951
5/1/2009,9:40:00 AM,18.29,18.38,18.25,18.37,53198
5/1/2009,9:45:00 AM,18.38,18.4,18.28,18.285,49491
And here is my error messages in SQL Management Studio:
Msg 102, Level 15, State 1, Line 4 Incorrect syntax near ','. Msg 208,
Level 16, State 82, Line 3 Invalid object name 'CSVTest1'. Msg 208,
Level 16, State 1, Line 3 Invalid object name 'CSVTest1'. Msg 3701,
Level 11, State 5, Line 3 Cannot drop the table 'CSVTest1', because it
does not exist or you do not have permission.
I would really appreciate help here, my head is about to explode after all these hours without any progression. I've tried MySQL too, didn't work there either.
As I'm new, I might need it explained to the details.
It appears you have an extraneous comma in the CREATE TABLE statement. There is a comma following the final column prior to the closing paren. Perhaps it is valid in some implementations, but you might try removing it. Change it to:
Volume varchar(10)
Ah - and it appears there is an extraneous closing parent in the second column definition. Change it to:
dateval smalldatetime,
And the time column:
timevale time,
Ultimately, it appears you should probably just try to get the CREATE TABLE statement syntax correct, then start adding the other parts.
There is no need for a comma after the last column definition: Volume varchar(10),.
I assume timevale should be timeval.
time() should just be time.
Also, I'm probably being picky but you have capitalised the first letter of all the column names except the first two - won't cause an error but probably better to have a consistent naming convention. I would capitalise the 'v' in val and write the whole word too.
The CSV data needs revising too - you need to specify EVERY column, even if it is null. See my example data (the new lines at the end of each row are for illustration purposes only).
1234567890,2012-08-25,22:15,anytext,ornum,for,varchar,columns <-new line
abcd123456,2010-05-20,00:01,anything,in,these,varchar,columns <-new line
abcd123456,2010-05-20,00:01,anything,in,,,columns <-new line
This works:
CREATE TABLE CSVTest1 (
Ticker varchar(10) NULL,
DateValue smalldatetime NULL,
TimeValue time NULL,
OpenValue varchar(10) NULL,
HighValue varchar(10) NULL,
LowValue varchar(10) NULL,
CloseValue varchar(10) NULL,
Volume varchar(10) NULL)
GO
BULK INSERT CSVTest1
FROM 'C:\TEST.txt'
WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\n')
GO
Your CSV file needs to have a new line for each record you want to insert, as specified by the ROWTERMINATOR = '\n' and a comma between each field as specified by FIELDTERMINATOR = ','.
EDIT:
By the way if you are using SQL Server Management Studio (SSMS) you can create the table through the user interface and then:
Right click on the table
Script Table as
CREATE To
New Query Editor Window