I need to run a query in SQL Server, where I have a particular number of values stored individually on separate lines in a text file, and I need to run a query in SQL server to check if a value in a column of the table matches any one of the value stored in the txt file.
How should I go about doing this ?
I am aware of how to formulate various types of queries in SQL Server, just not sure how to run a query that is dependent on a file for its query parameters.
EDIT :
Issue 1 : I am not doing this via a program since the query that I need to run traverses over 7 million datapoints which results in the program timing out before it can complete, hence the only alternative I have left is to run the query in SQL Server itself without worrying about the timeout.
Issue 2 : I do not have admin rights to the database that I am accessing which is why there is no way I could create a table, dump the file into it, then perform a query by joining those tables.
Thanks.
One option would be to use BULK INSERT and a temp table. Once in the temp table, you can parse the values. This is likely not the exact answer you need, but based on your experience, I'm sure you could tweak as needed.
Thanks...
SET NOCOUNT ON;
USE Your_DB;
GO
CREATE TABLE dbo.t (
i int,
n varchar(10),
d decimal(18,4),
dt datetime
);
GO
BULK INSERT dbo.t
FROM 'D:\import\data.txt'
WITH (FIELDTERMINATOR = ',', ROWTERMINATOR = '\n');
There are lots of approaches.
Mine would be to import the file to a table, do the comparison with a regular SQL query, and then delete the file-data table if you don't need it anymore.
Bulk import the data from text file into a temporary table.
Execute the query to do the comparison between your actual physical table & temporary table.
Related
I have 1 million rows of data in a file, I want to insert all the records into SQL Server. While inserting I am doing some comparison with existing data on the server, if the comparison satisfied I will update the existing records in the server or else I will insert the record from the file.
I'm currently doing this by looping from C#, which consume more than 3 hours to complete the work. Can anyone suggest idea to improve the performance?
Thanks,
Xavier.
Check if your database in Full or Simple recovery mode:
SELECT recovery_model_desc
FROM sys.databases
WHERE name = 'MyDataBase';
If database is SIMPLE recovery mode you can create a staging table right there. If it is in Full mode then better create Staging table in separate database with Simple model.
Use any BulkInsert operation/tool (for instance BCP, as already suggested)
Insert only those data from your staging table, which do not exist in your target table. (hope you know how to do it)
I have an old SQL script that is currently run by loading it into SQL Server Management studio and running it. I'd like to clean this up by turning it into a series of functions that are stored in the database itself.
The basic sequence of steps that the current code does is like this:
(Miles of SQL logic)
Create a temporary table
BULK INSERT from a CSV file into the temporary table
Massage the data
Merge the data into the "real" table
DROP the temporary table
(Miles of SQL logic)
I'd like to wrap steps 1-5 in a function, but I'm stuck at how to perform a BULK INSERT when you can't BULK INSERT into a table variable, and you're also not allowed to create temporary tables from within a function.
So what's the right way to fix this issue?
Thanks!
As already mentionned in the comment, the solution that differs the less to yours is doing that in a stored procedure rather than in a functoin, which is intended to modify the content of a table.
On a short term perspective, this should be clearly the easiest to implement for you but on a long term learnin SSIS could be a good investment.
I have a table which contains GUID and Name columns and I need to retrieve the last inserted rows so I can load it into table2.
But how would I find out the latest data in Table1. I seem to be lost at this I have read similar posts posing the same question but the answers don't seem to work for my situation.
I am using SQL Server 2008 and I upload my data using SSIS
1 - One way to do this is with triggers. Check out my blog entry that shows how to copy data from one table to another on a insert.
Triggers to replicate data = http://craftydba.com/?p=1995
However, like most things in life, there is overhead with triggers. If you are bulk loading a ton of data via SSIS, this can add up.
2 - Another way to do this is to add a modify date to your first table and modify your SSIS package.
ALTER TABLE [MyTable1]
ADD [ModifyDate] [datetime] NULL DEFAULT GETDATE();
Next, change your SSIS package. In the control flow, add an execute SQL task. Insert data from [MyTable1] to [MyTable2] using TSQL.
INSERT INTO [MyTable2]
SELECT * FROM [MyTable1]
WHERE [ModifyDate] >= 'Start Date/Time Of Package';
Execute SQL Task =
http://technet.microsoft.com/en-us/library/ms141003.aspx
This will be quicker than a data flow or execute OLEDB command since you are working with the data on the server.
I have a particular SQL file in which i copy all contents from on table in a database to another table in another database.
The traditional INSERT statements are used to perform the same operation. However this table has 8.5 Million records and it fails. The queries succeed with a smaller database.
Also in when i run the select * query for that particular table the SQL query express shows out of memory exception.
In particular there is one table that has some many records. So this table alone i want to copy from the old Db to the new Db.
What are alternate ways to achieve this?
Is there any quick work around by which we can avoid this exception and make the queries succeed?
Let me put it this way. Why would this operation fail when there are a lot of records?
I don't know if this counts as "traditional INSERT", but have you tried "INSERT INTO"?
http://www.w3schools.com/sql/sql_select_into.asp
Sorry if already asked, but I can't find anything on this.
I am moving something over from MySQL to SQL Server I want to have a .sql file create a database and tables within the database. After working out syntax kinks I have gotten the files to work (almost).
If I run
IF db_id('dbname') IS NULL
CREATE DATABASE dbname
it works fine, and if I run
CREATE TABLE dbname.dbo.TABLE1 (
);
...
CREATE TABLE dbname.dbo.TABLEN (
);
it also works fine. But, if I run them in the same file I get this error
Database 'dbname' does not exist
Right now, the CREATE TABLE statements are not within the IF statement, which I would like, but I also cannot seem to find the syntax for that. ( { } does not work?)
So my big question is, how do I ensure a particular command in a .sql file is completed before another in SQL Server?
My second question is, how do I include multiple instructions within an IF clause?
To be clear, I have been running this into sqlcmd.
Put a GO command between queries.
IF db_id('dbname') IS NULL
CREATE DATABASE dbname
GO
CREATE TABLE dbname.dbo.TABLE1 (
);
CREATE TABLE dbname.dbo.TABLEN (
);
As for putting the table statements in the IF, you wouldn't be able to because of the GO command. You could create additional IF statements afterwards, to check for each tables pre-existence.
The syntax for a block if is:
IF condition
BEGIN
....
....
END
Between creating the database and creating the tables you will need a USE statement.
USE dbname
This way the tables will be created in the correct place, without having to specify the DB name on everything.
Also, GO and BEGIN...END like everyone else is saying.
You have to separate the statements with the GO keyword:
sql query
GO
another sql query
GO
and so on
By placing a GO between statements (to create separate batches of statements)