I got a big file contains with 10000 lines table rows data, and I had to insert it with a SQL method in my project, so I generated a very long SQL after parsing it. But it doesn't allow me to execute this at one time.
I think it should be placed in a transaction, and I got the way to use transaction with one short SQL in this doc. What method should I use in its document?
Hope you could process at SQL Server side using Bulk insert or OLEDB
Related
I came across a good SSIS and SQL Problem. How do I in SSIS create a package that will execute a SQL query in management studio and grab the results of that query (the query results are "Insert INTO statements") and run that insert into statement query results into another sql database within SSIS that updates a table in another server? (The first query runs in one database and the second query runs in a different database)
First of all, sql queries execute on the database, not management studio. Management studio is visual interface for configuring,managing and administer databases.
To me it doesn't sound like there's any problem here at all. Create one connection manager for each DB. Then create two "Execute SQL Tasks", put your insert statements in them use use your connection manangers you've created.
Run the first query in an Execute SQL task and store the results in a string variable.
Then run a second Execute SQL task, using the variable as your SQL command.
Create Connection Managers for each of the databases you need, your source and both (or all) destinations.
Create a Data Flow Task.
In your OLEDB Source, execute your SELECT statement.
Pump the results into a MultiCast Transformation. This allows you to send the exact same result set to multiple destinations.
Create a Destination for each table you want to write to, and connect them to the MultiCast.
Bob's your uncle.
Thanks for answering...
I am new to SSIS and I have below requirement:
We are getting bulk records from Oracle based on some criteria.
I need to insert all these bulk records first to an audit table in SQL server The tables has only few columns. the reason for this is we will get to know how many records in each cycle we have got from oracle and if they are pilling up what is the exact reason.
After inserting in audit table, I need to process them one by one in SQL server and at the end of processing I would be either accepting the record and inserting the entire record in SQL serever or rejecting the record in SQL server based on business condition.
Currently to implement this I have first added a Data flow task in SSIS package which selects few columns(which are required by Audit table) from the Oracle and inserts them to SQL server after conversion. Than I am again getting the records from oracle using SQL execute task and processing them one by one using for each container in sql server.
As mentioned above I am making twice the call to Oracle, I am not able to reduce this to one?
can someone please guide me how can I achieve this?
Thanks in advance.
If you have the data already in SQL (you did a bulk copy in your audit table) you can take the data from there and do the insert in the table you want.
Instead of reading the data from Oracle twice, you can use your SQL Execute Task to manipule the data in SQL. You can make the insert statement from here to the final table.
I have a single SQL query that I need to run against ~25 different databases- each residing on a separate SQL server on the network. The query will run from a single central SQL server management studio, and the 24 other SQL server instances are linked. I have the query I need, and I tested that it works- however the goal is to create a script that queries each of the 25 separate SQL instances.
Instead of writing the query out 25 separate times within the script, I'm wondering if there's a way to utilize the single block of code to query each of the linked instances using an array, variables, DO/WHILE, a function or any other method.
Here's the query:
SET NOCOUNT ON
PRINT 'local server';
SELECT isc.ini_schema_name[Device], count(*) [Count]
FROM pharos.dbo.edi_pharos_stations eps, pharos.dbo.ini_schemas isc
WHERE eps.ini_schema_id = isc.ini_schema_id
GROUP BY isc.ini_schema_id, isc.ini_schema_name
For the purpose of this example, if I were to utilize the less-graceful approach of writing out the block of code 24 more times, this would be the next query in the script (to query SQL server hostnamed pharos90-2008).
PRINT 'Pharos90-2008';
SELECT isc.ini_schema_name[Device], count(*) [Count]
FROM [pharos90-2008].pharos.dbo.edi_pharos_stations eps, [pharos90-2008].pharos.dbo.ini_schemas isc
WHERE eps.ini_schema_id = isc.ini_schema_id
GROUP BY isc.ini_schema_id, isc.ini_schema_name
As you can see, the query / code is exactly the same except for the fact that it is referencing a separate linked SQL Server (query being run from a central SQL Server Management Studio).
The ultimate goal is to output the queried data for each SQL instance to a single .txt file; format being, print the name of each particular SQL server followed by the corresponding queried data.
Any advice as to how one would accomplish such a task?
Thanks in advance.
Well, one way would be to create a cursor to iterate all of your linked servers. (You can find linked servers like this...)
SELECT * FROM sys.servers WHERE is_linked = 1
Then, you could use the undocumented sp_MSForEachDB stored procedure to run a dynamic version of your query (changing the server on each iteration) on each database in the cursor's current server. If you search for sp_MSForEachDB you can find plenty of information. But here's one link to save time.
Is it possible to perform a bulk insert into an MS-SQL Server (2000, 2005, 2008) using the RODBC package?
I know that I can do this using freebcp, but I'm curious if the RODBC package implements this portion of the Microsoft SQL API and if not, how difficult it would be to implement it.
check out the new odbc and DBI packages. DBI::dbWriteTable writes around 20,000 records per second... Much much faster than the Row Inserts from RODBC::sqlSave()
You're probably looking for ?sqlSave which uses a parametrized INSERT INTO query (taking place in one operation) when you set Fast=True.
Now You can use dbBulkCopy from the new rsqlserver package:
A typical scenario:
You create a matrix
you save it as a csv file
You call dbBulkCopy to read fil and insert it using internally bcp tool of MS Sql server.
This assume that your table is already created in the data base:
dat <- matrix(round(rnorm(nrow*ncol),nrow,ncol)
id.file = "temp_file.csv"
write.csv(dat,file=id.file,row.names=FALSE)
dbBulkCopy(conn,'NEW_BP_TABLE',value=id.file)
Using RODBC, the fastest insert we've been able to create (260 million row insert) looks like the following (in R pseudo code):
ourDataFrame <- sqlQuery(OurConnection, "SELECT myDataThing1, myDataThing2
FROM myData")
ourDF <- doStuff(ourDataFrame)
write.csv(ourDF,ourFile)
sqlQuery(OurConnection, "CREATE TABLE myTable ( la [La], laLa [LaLa]);
BULK INSERT myTable FROM 'ourFile'
WITH YOURPARAMS=yourParams;")
If you're running this from between servers, you need a network drive that the R server can write to (e.g. one server with permissions for writing to the DB uses Rscript to productionalize the code), and the SQL Server can read from.
From everything I can find, there is NO solution for bulk insert to MySQL and nothing that works with SSIS which is why Microsoft is including in-database analytics with SQL Server 2016 after buying Revolution R Analytics.
I tried to comment on the previous answer but don't have the reputation to do it.
The rsqlserver package needs to run with rClr and neither of those packages are well-behaved, especially because rsqlserver's INSERT functions have poor data type handling. So if you use it, you'll have no idea what you're looking at in the SQL table as much of the information in your data.frame will have been transformed.
Considering the RODBC package has been around for 15 years, I'm pretty disappointed that no one has created a bulk insert function...
Our n2khelper package can use bcp (bulkcopy) when it is available. When not available it falls back to multiple INSERT statements.
You can find the package on https://github.com/INBO-Natura2000/n2khelper
Install it with devtools::install_git("INBO-Natura2000/n2khelper") and look for the odbc_insert() function.
I am using Excel to generate a whole slew of INSERT statements, similar to the process described here.
I would like to have an SSIS package that will run through each row of this excel file and run the SQL statements it finds against a database.
Is this possible?
EDIT:
As John points out - there is a better way to do it - generate the insert statement in SSIS instead of Excel. (I'm just getting my head into SSIS).
I'm sure it's possible, if a little silly. Why not generate the INSERT statements inside of SSIS?
At any rate, I suppose you'd get an Excel connection manager, use it in an Execute SQL Command task to SELECT the column with the INSERTS from Excel, and output the results to a Recordset. You'd then use a ForEach task to iterate over the recordset, putting each INSERT into a variable. Use the variable to configure an Execute SQL Task inside the ForEach task.