I can't seem to find the answer to this quite trivial question.
I would like to bulk import data from a .csv file (with an unknown number of columns, comma separated) file to a new SQL Server table, within an existing database. The BULK INSERT statement works fine if the table is predefined, but since I don't know the number of columns of the .csv file upfront, this won't work.
I was trying to use bulk in combination with openrowset, but can't get it working.
By the way: SSIS won't be an option in this case, since I would like to incorporate the query within R (sqlquery) or Python.
Help would be highly appreciated!
I have found a workaround, using R, to solve the problem above. The same logic can be applied in other languages. I advise everyone using this function to keep in mind the useful comments above.
I wrote a small function to capture the steps in R:
SQLSave <- function(dbhandle, data, tablename) {
# Export data to temp path, for example within your SQL Server directory.
write.csv2(data,file = "\\\\pathToSQL\\temp.csv",row.names=FALSE,na="")
# Write first 100 rows to SQL Server, to incorporate the data structure.
sqlSave(dbhandle, head(data,100), tablename = tablename, rownames = FALSE, safer = FALSE)
# SQL Query to remove data in the table, structure remains:
sqlQuery(dbhandle,paste("DELETE FROM [",tablename,"]",sep=""));
# SQL Query to bulk insert all data from temp .csv to SQL Server
sqlQuery(dbhandle,paste("BULK INSERT [",tablename,"]
FROM '\\\\pathToSQL\\temp.csv'
WITH
(
FIELDTERMINATOR = ';',
ROWTERMINATOR = '\\n',
FIRSTROW = 2,
KEEPNULLS
)",sep=""));
# Delete temp file from file directory
file.remove("\\\\pathToSQL\\temp.csv")
}
I am currently struggling with the same problem. I have first read the first row (headers) using bulk insert and created the table. Then again using bulk insert from row 2 imported data in the table. Although you have to change datatype after checking the data imported.
CREATE TABLE #Header(HeadString nvarchar(max))
declare #TableName nvarchar(100)='byom.DenormReportingData_100_100'
DECLARE #Columns nvarchar(max)=''
declare #Query nvarchar(max)=''
DECLARE #QUERY2 NVARCHAR(MAX)=''
bulk insert #Header
from 'F:/Data/BDL_BI_Test.csv'
with(firstrow=1,lastrow=1)
select #Columns=(select quotename(value)+' VARCHAR(500)'+',' from #Header cross apply string_split(HeadString,',') for xml path(''))
if isnull(#Columns,'')<>''
begin
set #Columns = left(#Columns,len(#Columns)-1)
select #Query=#Query+'CREATE TABLE '+#TableName+' ('+#Columns+')'
exec(#QUERY)
end
select #QUERY2 =#QUERY2+'bulk insert '+#TableName+' from ''F:/Data/BDL_BI_Test.csv''
with(firstrow=2,FORMAT=''csv'',FIELDTERMINATOR='','',ROWTERMINATOR=''\n'')'
exec(#QUERY2)
Related
Good morning; I currently receive my companies time logs on a daily basis, for the entire week. So for example it's Thursday now so I'll see entries for Monday (7/30) Tuesday (7/31) etc. I would like to just bulk load this into SQL but cannot seem to figure out how. If I use the flatfile import it loads it in in a snap no problems, but can only make a new table on import. I would prefer to just have this only update any new values, perhaps through a temptable and only load into my table "Timelog" the new values from the previous day. This is the flatfile import:
Which is exactly what I want, except two things. I know that even if I could get it to append to an existing table it'll reimport ALL the data again, which is a no go as well as I'll have duplicate values.
This is the CSV File
So I tried to write a bulkinsert statement:
BULK INSERT Timelog
FROM 'C:\Intel\555.csv'
WITH
(
FIRSTROW = 8,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'C:\CSVDATA\derrors.csv',
TABLOCK
)
This fails because it says it cannot find the file which I have to work on pathing, but I also know this won't work because it's not removing duplicate values. Can anyone help?
Either you have access problems. If you are on a SERVER and want to go to a direct path on your local PC, obvisouly you cant do that. Your file has to be on the same server as your SQL Server. Unless you specify anything else.
This example here worked for me:
declare #sql varchar(max)
select #sql = 'BULK INSERT dbo.Timelog FROM ';
select #sql = #sql +'''D:\Test\555csb.csv''';
select #sql = #sql + ' WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = '';'', --CSV field delimiter
ROWTERMINATOR = ''\n'', --Use to shift the control to next row
ERRORFILE = ''D:\Test\derrors.csv'',TABLOCK)';
print #sql
exec(#sql)
CSV File
Result
I have a requirement.
Using SSIS i am import data from flat file/excel file into my staging table. From staging table i need to filter data and transfer it to different databases over different linked server.i.e. let say for California i have dbCalifornia on Server A, For Taxes i have dbTaxes on Server B etc etc.
I need to read config table and redirect data accordingly.i.e. if column value =CALI insert data in dbCalifornia.tblA, for column value =TAX insert data in dbTaxes.tblA. I am trying to use Server Name and Database name as variable (because i am reading these from config table) i.e.
INSERT INTO [#server].[#database].[DBO].[BASIC]
But i am getting error .
I am not expert DBA please suggest my solution how can i implement this scenario.
TIA
You can do it using dynamic sql like this:
declare #server varchar(100), #database varchar(100);
DECLARE #sql varchar(8000) ='INSERT INTO[' + #server +'].['+ #database + '.[DBO].[BASIC]' +
'(EmpID,EmployeeID,ADDR1, ADDR2, ADDR3,ADDR4,TELNUM,MARRIED,LNAME,MNAME,FNAME,' +
'SEX, EMAIL,COUNTRYCODE, CITIZEN) ' +
'select EmpID,EmployeeID,ADDR1, ADDR2, ADDR3,ADDR4,TELNUM,MARRIED,LNAME,MNAME,FNAME,
SEX, EMAIL,COUNTRYCODE, CITIZEN from dbo.myExcelTable where state = ' + #database;
exec(#sql);
I don't understand what are your 100 variables tht you use in your insert,
didn't you say
if column value =CALI insert data in dbCalifornia.tblA, for column
value =TAX insert data in dbTaxes.tblA.
?
So you just need to filter your table using #database value and insert those rows in corresponding table
normally when reading a table from another server and DB. I use the LinkedServer, user and table like this:
Select * from Link..User.Table ;
after a linked server I have to use 2 dots.
I don't know if this can help since this is to read from an Oracle database.
I am working on a reporting project based in SQL but I have restricted access to the DB; I can only make SELECT Queries and insert the data I retrieve into temp Tables/table variables. I cannot create/execute stored procedures or any sort of functions.
The query I am running is meant to pool together all Engineers and the different key skills that they have so that we can later on see what Skills each engineer has or which Engineers fall under a certain skill.
To this end, I am trying to create a table variable/temp table with a flexible structure, a structure based on previously obtained values in the same query.
For E.g.
1st Output:
Adam
Brad
Julio
Martinez
2nd Output (Skill separated by white space):
VOIP
TTS
DBA
Exchange
Server
Create temp table/table variable that uses 1st output as rows and 2nd output as columns or vice versa. I will then populate this new table according to different values on the main DB.
Please advise how this can be done, or provide any other solution to this problem.
Thank you
I believe you can.
First of all you need to create temp table with dynamic structure based on query. It can be done like this:
declare script template:
Set #ScriptTmpl = 'Alter table #tempTable Add [?] varchar(100);
build script that will insert columns you need based on query:
Select #TableScript = #TableScript + Replace(#ScriptTmpl, '?',
ColumnName) From ... Where ...
then execute script and then fill your new table with values from second query
UPD:
here is the full sample of temporary table dynamic creation. I used global temporary table in my sample:
declare #scriptTemplate nvarchar(MAX)
declare #script nvarchar(MAX)
declare #tableTemplate nvarchar(MAX)
SET #tableTemplate = 'create table ##tmptable (?)'
SET #scriptTemplate = '? nvarchar(500),'
SET #script = ''
Drop table ##tmptable
Select #script = #script + Replace(#scriptTemplate, '?', [Name])
From Account
Where name like 'ES_%'
SET #script = LEFT(#script, LEN(#script) - 1)
SET #script = Replace(#tableTemplate, '?', #script)
Select #script
exec(#script)
Select * from ##tmptable
Firstly, you may be able to achieve what you want through pivots, rather than temporary tables.
Secondly, if you really want to create a table with column name "Adam Brad", the solution is dynamic SQL, which you may not be able to do based on your permissions.
I need som help with a problem, our company have a vendor that deliver a database to us. inside that database, the vendor has a table with alot of t sql scripts. What i want to do is the following, i want to make a select to find the script and then execut the script and store the result in a variable or temp tabel. I can not alter the script from the vedor, so I need the result into something i can manupilate. Another problem is that i dont know how many columns ther result will have. So it has to be flexible. Like one script have 5 columns and and the next script has 8 and so on.
exsample:
DECLARE #SQL nvarchar(MAX) = ( Select distinct script_details
from scripttable where .......)
This will give me the script I want to use, then I use
EXEC(#SQL)
to execute the script.
Then my problem is, the result from this I want into a variable or a table.
I have tryed to make a temp table like this:
create table #TmpTblSP (col1 varchar(MAX),col2 varchar(MAX),col3 varchar(MAX),col4 varchar(MAX),col5 varchar(MAX),col6 varchar(MAX),col7 varchar(MAX),col8 varchar(MAX),col9 varchar(MAX),col10 varchar(MAX),col11 varchar(MAX),col12 varchar(MAX))
then
insert into #TmpTblSP
EXEC(#SQL)
This gives me the following error:
Msg 213, Level 16, State 7, Line 1
Column name or number of supplied values does not match table definition.
But if i know how many columns there are and specify that into the insert it works.
insert into #TmpTblSP(Col1,Col2,Col3)
EXEC(#SQL)
But here you se my problem, I dont know how many columns there are in every script. I could make one script for every script the vendor has, but that will be alot, it's like 3000 scripts in that table and they change them often.
You could try something like:
DECLARE #SQL nvarchar(MAX) = (
Select distinct script_details
into #temptbl
from scripttable where .......
);
EXEC(#SQL);
If you don't know how many columns yous #sql gives then the only solution is use SELECT INTO. I use it in this way:
DECLARE #QRY nvarchar(MAX) = ( Select distinct script_details
from scripttable where .......)
SET #sql = 'SELECT * into ' + #temptablename + ' FROM (' + #qry + ') A '
It gives some flexibility
Remember that it is easy to check structure of the table created in this way in sys so you can build another #SQL from this info if needed.
I this as well recommended to split "SELECT INTO" to 2 parts
One is
SELECT INTO ......... WHERE 1=2
Second
INSERT INTO SELECT ......
Creation of table locks all DB. So it is good to create it as fast as possible and then insert into it.
So I am basing my code on the Import to mapped columns question asked by another user.
Here is the code...
DECLARE #TempTable TABLE (Name nvarchar(max))
BULK INSERT #TempTable
FROM ‘C:\YourFilePath\file.csv’
WITH ( FIELDTERMINATOR = ‘,’,
ROWTERMINATOR = ‘\n’
)
INSERT INTO TABLE ([Name], [TypeId])
Select Name,'99E05902-1F68-4B1A-BC66-A143BFF19E37' from #TempTable
Do I put this code into a stored procedure or a function to run it from my ASP script?
Is your TypeID a constant at the last line?
You can do either SP or TableFunction. But since TableFunction is depandant on the version of SQL Server you have, it might make sense to use SP.