Is it possible to perform a bulk insert into an MS-SQL Server (2000, 2005, 2008) using the RODBC package?
I know that I can do this using freebcp, but I'm curious if the RODBC package implements this portion of the Microsoft SQL API and if not, how difficult it would be to implement it.
check out the new odbc and DBI packages. DBI::dbWriteTable writes around 20,000 records per second... Much much faster than the Row Inserts from RODBC::sqlSave()
You're probably looking for ?sqlSave which uses a parametrized INSERT INTO query (taking place in one operation) when you set Fast=True.
Now You can use dbBulkCopy from the new rsqlserver package:
A typical scenario:
You create a matrix
you save it as a csv file
You call dbBulkCopy to read fil and insert it using internally bcp tool of MS Sql server.
This assume that your table is already created in the data base:
dat <- matrix(round(rnorm(nrow*ncol),nrow,ncol)
id.file = "temp_file.csv"
write.csv(dat,file=id.file,row.names=FALSE)
dbBulkCopy(conn,'NEW_BP_TABLE',value=id.file)
Using RODBC, the fastest insert we've been able to create (260 million row insert) looks like the following (in R pseudo code):
ourDataFrame <- sqlQuery(OurConnection, "SELECT myDataThing1, myDataThing2
FROM myData")
ourDF <- doStuff(ourDataFrame)
write.csv(ourDF,ourFile)
sqlQuery(OurConnection, "CREATE TABLE myTable ( la [La], laLa [LaLa]);
BULK INSERT myTable FROM 'ourFile'
WITH YOURPARAMS=yourParams;")
If you're running this from between servers, you need a network drive that the R server can write to (e.g. one server with permissions for writing to the DB uses Rscript to productionalize the code), and the SQL Server can read from.
From everything I can find, there is NO solution for bulk insert to MySQL and nothing that works with SSIS which is why Microsoft is including in-database analytics with SQL Server 2016 after buying Revolution R Analytics.
I tried to comment on the previous answer but don't have the reputation to do it.
The rsqlserver package needs to run with rClr and neither of those packages are well-behaved, especially because rsqlserver's INSERT functions have poor data type handling. So if you use it, you'll have no idea what you're looking at in the SQL table as much of the information in your data.frame will have been transformed.
Considering the RODBC package has been around for 15 years, I'm pretty disappointed that no one has created a bulk insert function...
Our n2khelper package can use bcp (bulkcopy) when it is available. When not available it falls back to multiple INSERT statements.
You can find the package on https://github.com/INBO-Natura2000/n2khelper
Install it with devtools::install_git("INBO-Natura2000/n2khelper") and look for the odbc_insert() function.
Related
After trying a few different packages and methods found online, I am yet to find a solution that works for inserting a dataframe from R into an existing table in SQL Server.
I've had great success doing this with MySQL, but SQL Server seems to be more difficult.
I have managed to write a new table using the DBI package, but I can't find a way to insert into using this method. Looking at the documentation, there doesn't seem to be a way of inserting.
As there are more than 1000 rows of data, using sqlQuery from the RODBC package also seems unfeasable.
Can anybody suggest a working method for inserting large amounts of data from a dataframe into an existing SQL table?
I've had similar needs using R and PostGreSQL using the r-postgres-specific drivers. I imagine similar issues may exist with SQLServer. The best solution I found was to write to a temporary table in the database using either dbWriteTable or one of the underlying functions to write from a stream to load very large tables (for Postgres, postgresqlCopyInDataframe, for example). The latter usually requires more work in terms of defining and aligning SQL data types and R class types to ensure writing, wheres dbWriteTable tends to be a bit easier. Once written to a temporary table, to then issue an SQL statement to insert into your table as you would within the database environment. Below is an example using high-level DBI library database calls:
dbExecute(conn,"start transaction;")
dbExecute(conn,"drop table if exists myTempTable")
dbWriteTable(conn,"myTempTable",df)
dbExecute(conn,"insert into myRealTable(a,b,c) select a,b,c from myTempTable")
dbExecute(conn,"drop table if exists myTempTable")
dbExecute(conn,"commit;")
I have to port sql server (2008) database with t-sql scripts. I can generate "create" script per each database object (stored procedure, table) from Sql Server Management Studio (though it looks to take much time)
How do I port data for tables? I'd like to have scripts like that:
INSERT INTO ... VALUES(...)
INSERT INTO ... VALUES(...)
INSERT INTO ... VALUES(...)
...
Can I generate such scripts from Sql Server Management Studio or is there some free 3'rd party utility for that? (I guess there should be).
Thank you in advance!
The (free) SMSS Tool pack addin can generate insert scripts for a DB.
If you're going to be doing bulk inserts of data, I'd suggest using bulk insert. You can do the insert from T-SQL, but I prefer to use the bcp command line utility as I can do both the export and import with minimal change to the run line. Oh... and it runs a lot faster than a bunch of insert statements. Have a look at the documentation and see if it fits your purposes.
I am using WebMatrix for a site right now, and its built-in SQL Server Compact database, and it's alright, but it only lets you create one row at a time. It has no bulk insert features (as I expected). But, see I have tens of thousands of rows in a spreadsheet.
I used to use Navicat for SQL Server which let me define a table name, then it would automatically IMPORT the spreadsheet into a table! Tens of thousands of rows, All within about 30seconds. How can I get Navicat for SQL Server to connect to WebMatrix's database for my website so I can do mass-bulk-inserts?
I have a Bulk Insert library, that you may be able to use: http://sqlcebulkcopy.codeplex.com
I've to INSERT a lot of rows (more than 1.000.000.000) to a SQL Server data base. The table has an AI Id, two varchar(80) cols and a smalldatetime with GETDATE as default value. The last one is just for auditory, but necesary.
I'd like to know the best (fastest) way to INSERT the rows. I've been reading about BULK INSERT. But if posible I'd like to avoid it because the app does not run on the same server where database is hosted and I'd like to keep them as isolated as posible.
Thanks!
Diego
Another option would be bcp.
Alternatively, if you're using .NET you can use the SqlBulkCopy class to bulk insert data. This is something I've blogged about on the performance of, which you may be interested in as I compared SqlBulkCopy vs another way of bulk loading data to SQL Server from .NET (using SqlDataAdapter). Basic example loading 100,000 rows took 0.8229s using SqlBulkCopy vs. 25.0729s using the SqlDataAdapter approach.
Create an SSIS package that will copy the file to SQL server machine and then use the data flow task to import data from file to SQL server database.
There is no faster/more efficient way than BULK INSERT and when you're dealing with such large ammount of data, do not even think about anything from .NET, because thanks to GC, managing millions of object in memory causes massive performance degradation.
How can I export all of my rows in a table to sql script in Microsoft SQL Server 2005 and then import them to another database?
Thanks in advance
If you moving it to another sql db you can right click the database you want and choose tasks -> generate scripts. That will launch a a wizard - follow along, choose the option to script all tables and data. Then execute that script in the new db(assuming that you've already created one with the same name)
If you can't find a data import/export tool that will work in your particular circumstances, it's possible to write plain SQL SELECT queries that will generate SQL INSERT statements. In this way it's possible to "export" all your data to a script file that can be run against the destination database. It's kind of an ugly hack, but it's simple and it works if you don't have a lot of data to move. See my answer to this question for details: Export SQL Server 2005 query result to SQL INSERT statement?
Note that this method assumes that the destination table already exists. But it's pretty straightforward to generate table creation scripts, as J Cory's answer has already shown.
There's a command line tool available to dump your data from particular tables into a SQL script that be executed against a different database:
http://blog.sqlauthority.com/2007/11/16/sql-server-2005-generate-script-with-data-from-database-database-publishing-wizard/
I don't believe SQL Management Studio Express supports data scripting (as your screenshot on J Cory's answer shows), but the full version does support that feature. In either case, the command line tool should accomplish what you need.