How do I perform a bulk insert of records with just one call/trip, using Microsoft Enterprise Library 5?
There is no clear cut method that the library provides to implement this as per my knowledge.
Following steps will help you implement bulk inserts using EntLib 5.0
-Implement a CustomTraceListener to override the TraceData method
-Pass the records in a serialized XML or something, as a param to the stored proc in DB
-Further processing can be done in the stored proc to read the XML and place each record into DB
This technique will avoid hitting the DB multiple times with just one call.
Hope it solves your problem.
Related
I've to INSERT a lot of rows (more than 1.000.000.000) to a SQL Server data base. The table has an AI Id, two varchar(80) cols and a smalldatetime with GETDATE as default value. The last one is just for auditory, but necesary.
I'd like to know the best (fastest) way to INSERT the rows. I've been reading about BULK INSERT. But if posible I'd like to avoid it because the app does not run on the same server where database is hosted and I'd like to keep them as isolated as posible.
Thanks!
Diego
Another option would be bcp.
Alternatively, if you're using .NET you can use the SqlBulkCopy class to bulk insert data. This is something I've blogged about on the performance of, which you may be interested in as I compared SqlBulkCopy vs another way of bulk loading data to SQL Server from .NET (using SqlDataAdapter). Basic example loading 100,000 rows took 0.8229s using SqlBulkCopy vs. 25.0729s using the SqlDataAdapter approach.
Create an SSIS package that will copy the file to SQL server machine and then use the data flow task to import data from file to SQL server database.
There is no faster/more efficient way than BULK INSERT and when you're dealing with such large ammount of data, do not even think about anything from .NET, because thanks to GC, managing millions of object in memory causes massive performance degradation.
I have two DB's one is feed by filtered data from another, now i'm using perl script witch executes query on foreign DB, stores a result in a csv file, and loads it to local DB using \COPY sytnatx
Is there a way to write plpgsql function witch will connect to foreign DB and load filtered data in local DB ( I know it can be done in ie. plperl, but i search more "native" way )
And there is the DBI-LINK that supports much more databases :)
Currently, PostgreSQL has dblink, but it only supports connecting to other PostgreSQL instances - not any other database, sadly.
I would recommend PL/Proxy, which is significantly easier to use - just write the desired stored procedure on the target database (with some minor caveats, like not using enumerated types), and declare the same function on the source, PL/Proxy will handle the communications. It is the basis for Skype's distributed database architecture and is production-ready.
I am using SQL server and ODBC in visual c++ for writing to the database. Currently i am using parameter binding in SQL queries ( as i fill the database with only 5 - 6 queries and same is true for retrieving data). I dont know much about stored procedures and I am wondering how much if any performance increase stored procedures have over parameter binding as in parameter binding we prepare the query only once and just execute it later in the program for diferent set of values of variables.
Stored procedures should be more performant for a few reasons:
Less network traffic - the query is on the DB and you just send a small command to the DB with params vs sending the entire query every time
The query is pre compiled on the server and can be cached as well by the DB
Another advantage is that you can alter the query on the DB without having to recompile the code. This is an additional layer of abstraction that I find very useful.
Is it possible to perform a bulk insert into an MS-SQL Server (2000, 2005, 2008) using the RODBC package?
I know that I can do this using freebcp, but I'm curious if the RODBC package implements this portion of the Microsoft SQL API and if not, how difficult it would be to implement it.
check out the new odbc and DBI packages. DBI::dbWriteTable writes around 20,000 records per second... Much much faster than the Row Inserts from RODBC::sqlSave()
You're probably looking for ?sqlSave which uses a parametrized INSERT INTO query (taking place in one operation) when you set Fast=True.
Now You can use dbBulkCopy from the new rsqlserver package:
A typical scenario:
You create a matrix
you save it as a csv file
You call dbBulkCopy to read fil and insert it using internally bcp tool of MS Sql server.
This assume that your table is already created in the data base:
dat <- matrix(round(rnorm(nrow*ncol),nrow,ncol)
id.file = "temp_file.csv"
write.csv(dat,file=id.file,row.names=FALSE)
dbBulkCopy(conn,'NEW_BP_TABLE',value=id.file)
Using RODBC, the fastest insert we've been able to create (260 million row insert) looks like the following (in R pseudo code):
ourDataFrame <- sqlQuery(OurConnection, "SELECT myDataThing1, myDataThing2
FROM myData")
ourDF <- doStuff(ourDataFrame)
write.csv(ourDF,ourFile)
sqlQuery(OurConnection, "CREATE TABLE myTable ( la [La], laLa [LaLa]);
BULK INSERT myTable FROM 'ourFile'
WITH YOURPARAMS=yourParams;")
If you're running this from between servers, you need a network drive that the R server can write to (e.g. one server with permissions for writing to the DB uses Rscript to productionalize the code), and the SQL Server can read from.
From everything I can find, there is NO solution for bulk insert to MySQL and nothing that works with SSIS which is why Microsoft is including in-database analytics with SQL Server 2016 after buying Revolution R Analytics.
I tried to comment on the previous answer but don't have the reputation to do it.
The rsqlserver package needs to run with rClr and neither of those packages are well-behaved, especially because rsqlserver's INSERT functions have poor data type handling. So if you use it, you'll have no idea what you're looking at in the SQL table as much of the information in your data.frame will have been transformed.
Considering the RODBC package has been around for 15 years, I'm pretty disappointed that no one has created a bulk insert function...
Our n2khelper package can use bcp (bulkcopy) when it is available. When not available it falls back to multiple INSERT statements.
You can find the package on https://github.com/INBO-Natura2000/n2khelper
Install it with devtools::install_git("INBO-Natura2000/n2khelper") and look for the odbc_insert() function.
I need:
Plain SQL that I can run without modification with sqlcmd.exe to insert testdata into testdatabase.
I have:
Service calls and entities to generate the insert operations with NHibernate.
Not working solution:
Log output to text-file. NHibernate generates parameterized sql but logs them in a format not runnable by sqlcmd.exe.
Is there any way to force NHibernate to generate sql without parameters?
Or is there any better solutions to the problem?
depending on your schema, it may be easier to just generate INSERTs from the actual database, try using a utility like:
Procedure to script your data (to generate INSERT statements from the existing data)
You could record a transaction log, the SQL Server Profiler is providing something like this.
In our application, we wrote factories in C# which generate the entities. We don't have SQL scripts to create testdata. An executer (.exe) picks up assemblies, creates the entities and stores them to the db. This way we don't have to maintain scripts. The factories are compile-time-safe.