I am trying to copy a table in a database into another database on another connection in VB.NET, using OleDb. If they were on the same connection I would just use SELECT INTO, but they are not. I have two different OleDbConnection and cannot see an easy way to do this.
Right now I am attempting to just copy the database into a DataTable using an OleDbDataAdapter, and then loop through the DataTable and insert every record into the target database one at a time. This obviously takes a ton of time for the large DB I could potentially be dealing with, and I have to deal with escaping strings, null values, etc.
Is there an easier way to do this?
Thanks,
Logan
edit - just to make this more clear: I have two OleDbConnection objects, one is linked directly to a local .mdb file on my computer (JET). The other is linked to a database on our servers (SQLOLEDB). I am wanting to do this:
"SELECT * FROM fromDB INTO toDB"
But I can't because fromDB and toDB are on different connections, and the OleDbCommand object is only attached to one. The only way I can see how to do this is to connect to fromDB, copy it into a DataTable, connect to toDB, and copy all of the data in the DataTable row by row into toDB. I was wondering if there is an easier way to do this.
If you are constrained to this architecture, one idea is to write a stored procedure on the server that accepts a large chuck of row data in one call. It could then write out the row data to a file for a future bulk-insert, or it could attempt to insert the rows directly.
This also has the benefit of speeding things up over high latency connections to the server.
Also, if you use parameterized statements, you can avoid having to escape strings etc.
If you are just copying from one to the other, why don't you do it in SQL?
You can create a Synonym within one database pointing at a table, view or stored proc on another database (on another server). You can then insert into this synonym just like you could into a table in the same db.
http://www.developer.com/db/article.php/3613301/Using-Synonyms-in-SQL-Server-2005.htm
Related
I am new to access. I am using a tool/access database someone built, and it has an ODBC connection to an Oracle SQL database in it.
There are different queries on the side panel, and some of them are delete queries.
If I run these delete queries will they just modify data in my local access database without modifying the data in the Oracle Database?
Yes these will change something in the database whether its linked with another access database table or oracle table and within the database. To review the query you can open the queries in design view and run a normal select query so you can see what the queries are deleting. You can have a normal table image and or globe with a arrow in front pointing towards the table then its linked. A lot of times when I am testing I just run select queries and then I make a copy of what I will be deleting just in case anything goes wrong.
Let say, I am connected to two different databases (one SQLite and one Oracle) over ODBC in one program. Is it possible to execute a query on one database and insert the resultset as a new table in the second database directly by just passing something like a data cursor, i.e. without the hurdle to create insert statements with explicit values from the result set and executing those on the destination database?
As far as I am aware you cannot do that. You can use something like a join engine to do it (e.g., http://www.easysoft.com/products/data_access/odbc_odbc_join_engine/index.html) but it might be overkill if you are doing it just the once.
If you use Oracle then you can use Oracle Heterogeneous Services which can work with ODBC sources. Have a look at: http://www.dba-oracle.com/t_heterogeneous_database_connections_sql_server.htm
I've got about 13,000 AccountIDs that I need to pull various data from several data tables using left-joins. The total # of accountIDs is in the millions. I don't have write-access to the server but I was wondering if there was a way I could maybe create a custom/temporary table anyway and do a join to that rather than writing a really, really long Where AccountID in (.....) statement. The accountIDs are currently in a single Excel column so I'd have to get them back in the server somehow.
Thoughts?
You can use OPENDATASOURCE to access your Excel file, but someone will have to push you the Excel file to the server (not ideal).
SELECT * FROM OPENDATASOURCE('Microsoft.Jet.OLEDB.4.0',
'Data Source=C:\DataFolder\Documents\TestExcel.xls;Extended Properties=EXCEL 5.0')...[Sheet1$] ;
You can ask your DBA team to install a SQl Server Client on your local machine and set up a linked server to the live server, and you are all set. Ask them to set up a view of the data you need, and give you access to the view. That's all you need, I guess.
I have nearly 7 billion rows of data in memory (list<T> and sortedlist<T,T>) in C#. I want to insert this data into tables in SQL Server. To do this, I define different SqlConnection for each collection and set connection pool to False.
First, I tried to insert data with connected mode (ExecuteNonQuery). Even I defined Parallel.Invoke and called all insert methods for different collections concurrently, it is too slow and up to now I couldn't finish it (I couldn't discriminate any differences between sequential and concurrent insert).
Also, I tried to create an object from SqlDataTable. To fill tables I read all data from collections once and add data to SqlDataTable. In this case I set SqlBatchSize=10000 and SqlTimeOut=0 for SqlBulkCopy. But this one also is very slow.
How can I insert a huge amount of data into SQL Server fast?
Look for 'BULK INSERT'. The technique is available for various RDBMS. Basically, you create a (text)file with one line per record and tell the server to consume this text file. This is the fastest approach I could think of. I import 50 million rows in a couple of seconds that way.
You already discovered SqlBulkCopy but you say it is slow. This can be because of two reasons:
You are using too small batches. Try to stream the rows in using a custom IDataReader that you pass to WriteToServer (or just use bigger DataTables)
Your table has nonclustered indexes. Disable them pre-import and regenerate them
You can't go faster than with bulk-import, though.
I've got a database that resides on an SQL server box, and another in a separate mdb file.
Both contain similar data, but I'd like to run a query that checks unmatched records from a field that exists in both.
Is this something that's easy enough to do using ADO (VBA)? If so can you point me in the right direction?
The easiest would be to create a new Access Database or ADP, and link the tables from both the SQL server and the other MDB. That way you've got an interface to both from within the same instance, which allows you to query or join different tables.