I have 1 million rows of data in a file, I want to insert all the records into SQL Server. While inserting I am doing some comparison with existing data on the server, if the comparison satisfied I will update the existing records in the server or else I will insert the record from the file.
I'm currently doing this by looping from C#, which consume more than 3 hours to complete the work. Can anyone suggest idea to improve the performance?
Thanks,
Xavier.
Check if your database in Full or Simple recovery mode:
SELECT recovery_model_desc
FROM sys.databases
WHERE name = 'MyDataBase';
If database is SIMPLE recovery mode you can create a staging table right there. If it is in Full mode then better create Staging table in separate database with Simple model.
Use any BulkInsert operation/tool (for instance BCP, as already suggested)
Insert only those data from your staging table, which do not exist in your target table. (hope you know how to do it)
Related
I have a SQL Server database, multi-user can insert to it.
But for many reasons, I want only 1 user can insert at a time.
Example:
User 1 want to insert 100 record, while user 1 is inserting (100 record not saved to table). Other user can not insert to the table.
I have thought to use a flag, but I want to find another way.
Is there any SQL statement that can do that?
Thanks for reading!
It seems that you need to use
INSERT INTO TABLE with (rowlock)
Read the following post to have a better understanding.
Using ROWLOCK in an INSERT statement (SQL Server)
Updated
SQL supports us to handle 1 record at a time e. And your case is to want multiple records to handle serialized format.
I think the best you put into the temp table, there is a window service running real-time (Background service: using quartz job or hangfire): insert and delete then the temporary table with a column named IsInserted.
For that purpose you can used table lock or row loack concept.
ALTER TABLE Table_name SET (LOCK_ESCALATION = < TABLE | AUTO | DISABLE > –One of those options)
For more details you can also visit this link
locking in SQL Server
I need to run a query in SQL Server, where I have a particular number of values stored individually on separate lines in a text file, and I need to run a query in SQL server to check if a value in a column of the table matches any one of the value stored in the txt file.
How should I go about doing this ?
I am aware of how to formulate various types of queries in SQL Server, just not sure how to run a query that is dependent on a file for its query parameters.
EDIT :
Issue 1 : I am not doing this via a program since the query that I need to run traverses over 7 million datapoints which results in the program timing out before it can complete, hence the only alternative I have left is to run the query in SQL Server itself without worrying about the timeout.
Issue 2 : I do not have admin rights to the database that I am accessing which is why there is no way I could create a table, dump the file into it, then perform a query by joining those tables.
Thanks.
One option would be to use BULK INSERT and a temp table. Once in the temp table, you can parse the values. This is likely not the exact answer you need, but based on your experience, I'm sure you could tweak as needed.
Thanks...
SET NOCOUNT ON;
USE Your_DB;
GO
CREATE TABLE dbo.t (
i int,
n varchar(10),
d decimal(18,4),
dt datetime
);
GO
BULK INSERT dbo.t
FROM 'D:\import\data.txt'
WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\n');
There are lots of approaches.
Mine would be to import the file to a table, do the comparison with a regular SQL query, and then delete the file-data table if you don't need it anymore.
Bulk import the data from text file into a temporary table.
Execute the query to do the comparison between your actual physical table & temporary table.
I want to insert data from database1 into database2 when something changes. Like a trigger but the thing is I am not allowed to create a trigger in database1.
I basically need to insert certain data into a table from a database into another database as they happen.
Any suggestions would be great!
Thanx
You can create a program to do that, there are many ways instead but I think this is the straightforward and easy one.
For example if you work with SQL server you can create C# code that connect to the two databases and check for data in the first one then send it to the second.
For example:
you can create a trigger for the first DB tables you want to check them, then you can create a web service that check if the trigger fire, it will get the data from first database and send it to the second, this will be better to enhance and you can change the code to do whatever you want without making any changes from the database side
MS SQL Server Replication
I think suitable for you is Transactional Replication
Transactional replication is asynchronous but close to real time
What about the merge operator ?
Merge operator will commit changes into the target table from the source table when something change.
As you cannot use trigger this cannot be a real time process but with a process scheduler like SQL Agent, you can run it each 10 seconds depending on your table size.
https://msdn.microsoft.com/en-us/library/bb510625.aspx
You can do it when your source come from multiple table by using CTE like that :
With TableSources As
(
Select id,Column1, Column2 from table1
UNION
Select id,Colum1,Column2 from table2
)
MERGE INTO TargetTable
USING TableSources ON TargetTable.id=TableSources.id
WHEN NOT MATCHED BY Target
Insert(Column1,Column2) Values(Column1,Column2)
WHEN NOT MATCHED BY Source
Delete
WHEN MATCHED
Update Set Column1=TableSources.Column1, Column2=TableSources.Column2
;
I have a person table (sqlServer 2008r2) and are using .vbs and .bat files to sync some data from a table called person in database A to database B.
Note - The DB names are different but the table names are the same. Because the peson table has 137 fields I am looking for a way to write both an INSERT and an UPDATE statement. How can I do thsi without listing 137 fields?
At the moment I connect to database A and populate a recordSet called RS with the people table records.
Then I loop through RS and query the people table in database B
If found I update people table in database B with the people table record from database A
If not found I insert the people record from database A into the people table on database B
Now this is nice and easy but since it has 137 fields I do not want to write a MASSIVE update and insert statements. DO I have another option such as:
The table structures are identical between people on database A and people on database B but the recordset obtained in step 1 above is using one one DB connection and the query in step 2 is using a separate DB connection to a different DB instance on a different server.
Help.
You can do that by using a fully qualified name.
insert into DATABASE1.dbo.Person (columns)
select (columns) from DATABASE2.dbo.Person
Also there are others way to pump data from a database for another
You can take a look on your MSSMS it got a tool for that kind of thing.
Select one database and right button mouse to open the import/export wizard.
You can also take a look on replication, bulk copy, service broker, etc...
I have a particular SQL file in which i copy all contents from on table in a database to another table in another database.
The traditional INSERT statements are used to perform the same operation. However this table has 8.5 Million records and it fails. The queries succeed with a smaller database.
Also in when i run the select * query for that particular table the SQL query express shows out of memory exception.
In particular there is one table that has some many records. So this table alone i want to copy from the old Db to the new Db.
What are alternate ways to achieve this?
Is there any quick work around by which we can avoid this exception and make the queries succeed?
Let me put it this way. Why would this operation fail when there are a lot of records?
I don't know if this counts as "traditional INSERT", but have you tried "INSERT INTO"?
http://www.w3schools.com/sql/sql_select_into.asp