I am trying to do a batch insert in R using RJDBC. It seems like it inserts 1 row at a time which takes a lot of time.
I was wondering if anyone knows of a solution in R to do bulk insert data from R to SQL. I know RODBC can do parametrized insert which is fast but not as fast as bulk insert.
I don't know about your "R" language, but there is a BULK sql statement available in sqlExe.
sqlExe is a utility that connects to SQL databases via ODBC and will execute any valid SQL, plus it has some additional features ( http://sourceforge.net/projects/sqlexe/ )
For example, assuming the target table is:
table: [mydata]
-------------------
row_id char(1)
row_idx integer
row_desc char(32)
To do your insert task with sqlExe you would prepare a file with your input:
input.dat
a,1,this is row 1
b,2,this is row 2
c,3,this is row 3
d,4,this is row 4
The command line to import:
sql --dsn MYDB -e "BULK INSERT input.dat, INSERT INTO mydata(row_id,row_idx,row_desc) VALUES(?,?,?)"
Related
I need to export/script two tables to my local database to a remote anyway.
Anyway I cannot export them as INSERT INTO scripts because I cannot drop them in the remote database and populate them again (because of FK and integrity constraints). So, is it possibile to script tables as UPDATE statements for each row, instead of INSERT INTO? I'm using SQL Server 2008/2012
CREATE TABLE mytable(
ExtractTypeNum INTEGER NOT NULL --PRIMARY KEY
,FileOrderNum VARCHAR(11)
,PrevFileOrderNum VARCHAR(11)
,NextFileOrderNum VARCHAR(11)
,rownum1 INTEGER
,Statusflag1 VARCHAR(9)
);
INSERT INTO mytable(ExtractTypeNum,FileOrderNum,PrevFileOrderNum,NextFileOrderNum,rownum1,Statusflag1)
VALUES (1,'2016-09-191',NULL,'2016-09-192',1,'IsInitial');
INSERT INTO mytable(ExtractTypeNum,FileOrderNum,PrevFileOrderNum,NextFileOrderNum,rownum1,Statusflag1)
VALUES (2,'2016-09-192','2016-09-191','2016-09-201',2,NULL);
INSERT INTO mytable(ExtractTypeNum,FileOrderNum,PrevFileOrderNum,NextFileOrderNum,rownum1,Statusflag1)
VALUES (3,'2016-09-201','2016-09-192','2016-09-211',3,NULL);
select 'Update Table Xyz Set Abc='+Convert(varchar(25),rownum1)+' ' as X,*
from myTable
There is many ways to sync data between databases. As my experience, you can do by 2 main ways:
Using MERGE statement (it supports from mssql 2k8: https://msdn.microsoft.com/en-us/library/bb510625.aspx)
write dynamic query to generate data script for 2 target tables:
Select 'select * into #tmp from dbtarget.tbla where 1<0;' as data
Union
Select 'insert into #tmp values(' + convert(varchar, cola) + ', ' + convert(varchar, colb) + ', ' + convert(varchar, colc) + ');' as data from dbsource.tbla;
run the query above at database source to get output script data.
apply script data to target database.
using MERGE statement to merge data between #tmp table and target table.
Utilize Log Shipping feature to sync data between databases in the same structure and in one domain network.
https://msdn.microsoft.com/en-us/library/ms190640(v=sql.110).aspx
Many thanks to Alfaiz Ahmed this is my final working script:
select 'UPDATE ERGO.DBO.RESIDENZE SET CODISEDERESI='+CONVERT(varchar(50),CODISEDERESI)+
',DESCRIRESIDE='+isnull(CONVERT(varchar(100),''''+replace(DESCRIRESIDE,'''','''''')+''''),'''''')+
',INDIRIRESIDE='+isnull(CONVERT(varchar(100),''''+replace(INDIRIRESIDE,'''','''''')+''''),'''''')+
',NOMERESIDENZ='+isnull(CONVERT(varchar(50),''''+replace(NOMERESIDENZ,'''','''''')+''''),'''''')+
',VIARESIDENZA='+isnull(CONVERT(varchar(50),''''+replace(VIARESIDENZA,'''','''''')+''''),'''''')+
',CAPRESIDENZA='+isnull(CONVERT(varchar(50),''''+CAPRESIDENZA+''''),'''''')+
',CITTARESIDEN='+isnull(CONVERT(varchar(50),''''+replace(CITTARESIDEN,'''','''''')+''''),'''''')+
',EMAILRESIDEN='+isnull(CONVERT(varchar(100),''''+EMAILRESIDEN+''''),'''''')+
' WHERE CODICERESIDE='+CONVERT(varchar(50),CODICERESIDE)
from RESIDENZE
I had to use that replace() function because many Italian names have a single quote in their names so, for example, I wanted string D'AZEGLIO to become D''AZEGLIO in order to be correctly processed by SQL. Finally, before execute the query I press CTRL+SHIFT+F to save the output to a sql file as a generated script.
I'm new to SQL and I'm attempting to do a bulk insert into a view, however, when I execute the script the message says (0 row(s) affected).
This is what I'm executing:
BULK INSERT vwDocGeneration
FROM '\\servername\Data\Doc\Test.csv'
WITH
(
Fieldterminator = '|',
Rowterminator = '\r\n'
)
I've confirmed the row terminators in my source file and they end with CRLF. The view and the file being imported have the same number of columns. I'm stuck! Any ideas would be greatly appreciated!
Per Mike K 's suggestion I started looking at key constraints and after I adjusted on of them I was able to use the bulk insert! FYI I did insert into the view because the table had an additional field that wasn't included in my CSV file. Thanks for confirming its possible #Gordon Linoff.
If you are looking for the number of rows affected by that operation, then use this.
DECLARE #Rows int
DECLARE #TestTable table (col1 int, col2 int)
// your bulk insert operation
SELECT #Rows=##ROWCOUNT
SELECT #Rows AS Rows,##ROWCOUNT AS [ROWCOUNT]
Or you can first bulk insert into a table, then create an appropriate view from that table.
Following article might be useful -
http://www.w3schools.com/sql/sql_view.asp
http://www.codeproject.com/Articles/236425/How-to-insert-data-using-SQL-Views-created-using-m
What i want is to store java statement with a semi colon in my embedded derby database. However i am not quite able to generate the simple SQL insert statement that would work. I believe that when the control encounters the semi colon statement in the string if considers it as the termination character and errors are thrown. Try the below:
create table dummy(keys int, vals varchar(255));
insert into dummy values (10,'System.out.println();');
The above statement fails with errors. However if i remove the semi colon it works
insert into dummy values (10,'System.out.println()');
Has anyone seen this before if so which escape character did you use.
Thanks.
It may help to see what context you are trying to execute the sql statement in, i.e. the surrounding code. If this is in a java program, something like this may work:
int n = stmt.executeUpdate("insert into dummy values (10, 'System.out.println()$;') {ESCAPE '$'}");
Your statements work fine when I run them using Derby's "ij" sql editor:
ij version 10.12
ij> connect 'jdbc:derby:tt;create=true';
ij> create table dummy(keys int, vals varchar(255));
0 rows inserted/updated/deleted
ij> insert into dummy values (10,'System.out.println();');
1 row inserted/updated/deleted
ij> select * from dummy;
KEYS |VALS
--------------------------------------------------------------------------------
------------------------------------------------------------
10 |System.out.println();
1 row selected
Perhaps the problem lies elsewhere. If you could include more details about the exact message you get, that might help.
I am using SQL Server 2012
The query is:
drop table x
create table x(id int primary key)
insert into x values(5)
insert into x values(6)
begin tran
insert into x values(1),(2),(3),(3),(4)--Primary key violation
commit tran
select* from x
This returns
5
6
and another query
drop table x
create table x(id int primary key)
insert into x values(5)
insert into x values(6)
begin tran
insert into x values(1)
insert into x values(2)
insert into x values(3)
insert into x values(3) --Primary key violation
insert into x values (4)
commit tran
select * from x
This returns
1
2
3
4
5
6
So what is the difference in inserting values in SQL Server?
Between those 2 queries and why the different result sets?
Sample 1 has a single insert statement for the 1,2,3,3,4,5. This is a "bulk" insert statement (however SQL Server uses the term bulk insert in a different fashion). Essentially it means all the inserts in this line are executed as 1 single action.
Sample 2 has separate insert statements. Since there is no exception handling in place, there is no reason for the transaction to abort. The error is ignored, the other records are added, and the result is then what you see.
SQL server executes the queries as batches. So if any error occurs in the batch, according to MSDN, one the following is possible.
No statements in the batch are executed.
No statements in the batch are executed and the transaction is rolled back.
All of the statements before the error statement are executed.
All of the statements except the error statement are executed.
In your first case, "No statements in the batch are executed". And in your second case, "All of the statements except the error statement or executed".
For more about SQL batches, please refer the following MSDN articles,
Batches of SQL Statements
Executing Batches
Errors and Batches
I have SQL server procedure that will take one id and process and make some inserts and this is happening in sequential. How can I do it in parallelly?
WHILE (#InnerCount <= (SELECT Count(Id) FROM #MB))
BEGIN
SELECT #StartDate=StartDate,#EndDate=EndDate,#WeekMonthName=MonthName
FROM #MB Where Id=#InnerCount
DELETE FROM facttemp
Where ScoreCardId=#ScoreCardId AND StartDate=#StartDate AND EndDate=#EndDate
TRUNCATE TABLE #temp
INSERT INTO #temp (EmployeeId,EmployeeInstanceId,UserId,ScoreCardId,ScorecardTarget,OverAllScore,Rank,MetricId,MetricName,MetricScore,MetricTarget,MetricWeightPercent,MetricBandNumber,IsQualityMetric)
EXEC [dbo].[usp_ScoreCardPreSummaryWeeklyMonthlyData] #ScoreCardId,#StartDate,#EndDate,#AccountId,#AccountInstanceId
INSERT INTO facttemp (
EmployeeId,EmployeeInstanceId,UserId,ScoreCardId,ScorecardTarget,OverAllScore,Rank,
MetricId,MetricName,MetricScore,MetricTarget,MetricWeightPercent,MetricBandNumber,IsQualityMetric,
StartDate,EndDate,Month,Year,AccountId,AccountInstanceId)
SELECT EmployeeId,EmployeeInstanceId,UserId,
ScoreCardId,ScorecardTarget,OverAllScore,Rank,
MetricId,MetricName,MetricScore,MetricTarget,MetricWeightPercent,MetricBandNumber,IsQualityMetric,
#StartDate,#EndDate,#WeekMonthName, YEAR(#StartDate),#AccountId,#AccountInstanceId
FROM #temp
SET #InnerCount=#InnerCount+1
END --End of Monthly Lop
END
Well, what I would do in this case is creating a temporary SQL job (msdb.dbo.sp_add_job) with this T-SQL script in step 1 (msdb.dbo.sp_add_jobstep) and deleting the temporary SQL job in step 2 (msdb.dbo.sp_delete_job).
That way you can execute the temporary SQL job (msdb.dbo.sp_start_job) and continu running the script while the temporary SQL job is running. You do not have to worry about the temporary SQL jobs as they all delete themselves in step 2.