So this is going to be an odd question but I'm going to try and explain it as best as I can in order to assist anybody trying to help me here...
I am presented with a situation in which I am trying to copy data from one database to another to similar tables, however there is a slight difference which makes a world of difference. db1.table1 allows null values in col3 and does in fact have a number of rows which have null values but db2.table1 does not allow null values in col3 but I still need to copy the values over. Furthermore, db1.table1.col3 is a GUID while db2.table1.col3 is VARCHAR which is part of the issue. If db1.table1.col3 weren't of type GUID I was simply going to UPDATE the column with the text that I need to insert there that I am going to need in db2.table1.col3.
So, to summarize: I am looking for a way to
INSERT INTO db2.table1 (col1, col2, col3...) SELECT col1, col2, col3... FROM db1.table1 but at places where col3 is null, I need to insert text/varchar so that it's not null.
Is there any simpler way to do this than building a temporary table that anybody knows of?
use COALESCE or ISNULL with the replacement text that you want
for example ISNULL(Col3, 'Sometext')
for a GUID, you can use the NEWID() function since you can't insert regular text into a uniqueidentifier data type
The NEWID() function returns a GUID....for example
SELECT NEWID()
26C064EF-0AB6-4DBE-91B3-C2EE40DE7AD6
Related
The (normal) INSERT sytnax looks like this (disregarding personal choices of line breaks / parenthesis placements):
INSERT INTO MyTable (
Col1,
Col2,
...
) VALUES (
#value1,
#value2
...
)
Since the association between the column and the value is based on the index in two different lists, as it were, this can become unwieldy if you have many columns. If you need to change something, you might end up changing the wrong column, or you risk inserting the value for Col15 in Col16 and vice versa because both were of compatible types and thus the query runs without problems.
I have previously "worked around" it like this:
INSERT INTO MyTable ( Col1, SomeCol2, Col3WithLongName, Col4, ...)
VALUES (#col1, #someCol2, #col3WithLongName, #col4, ...)
This, of course, has its own problems in that the lines can become very long (and it's not particularly fun to manually align code).
Ideally I'd like an UPDATE-like syntax for INSERT, which AFAIK isn't possible but would look something like this:
INSERT INTO MyTable (
Col1 = #value1,
Col2 = #value2,
...
)
Here, the association between column and value is immediately clear. I know there are often several different syntaxes for the same thing in T-SQL, but I haven't found an INSERT syntax that has this quality (clear column/value mapping).
Is there an INSERT syntax that has this quality, at least to a larger extent?
(If it's relevant, I'm mostly wondering about this in the context of MERGE.)
I have a .csv file with 600 million plus rows. I need to upload this into a database. It will have 3 columns assigned as primary keys.
I use pandas to read the file in chunks of 1000 lines.
At each chunk iteration I use the
INSERT INTO db_name.dbo.table_name("col1", "col2", "col3", "col4")
VALUES (?,?,?,?)
cursor.executemany(query, df.values.tolist())
Syntax with pyodbc in python to upload data in chunks of 1000 lines.
Unfortunately, there are apparently some duplicate rows present. When the duplicate row is encountered the uploading stops with an error from SQL Server.
Question: how can I upload data such that whenever a duplicate is encountered instead of stopping it will just skip that line and upload the rest? I found some questions and answers on insert into table from another table, or insert into table from variables declared, but nothing on reading from a file and using insert into table col_names values() command.
Based on those answers one idea might be:
At each iteration of chunks:
Upload to a temp table
Do the insertion from the temp table into the final table
Delete the rows in the temp table
However, with such a large file each second counts, and I was looking for an answer with better efficiency.
I also tried to deal with duplicates using python, however, since the file is too large to fit into the memory I could not find a way to do that.
Question 2: if I were to use bulk insert, how would I achieve to skip over the duplicates?
Thank you
You can try to use a CTE and an INSERT ... SELECT ... WHERE NOT EXISTS.
WITH cte
AS
(
SELECT ? col1,
? col2,
? col3,
? col4
)
INSERT INTO db_name.dbo.table_name
(col1,
col2,
col3,
col4)
SELECT col1,
col2,
col3,
col4
FROM cte
WHERE NOT EXISTS (SELECT *
FROM db_name.dbo.table_name
WHERE table_name.col1 = cte.col1
AND table_name.col2 = cte.col2
AND table_name.col3 = cte.col3
AND table_name.col4 = cte.col4);
Possibly delete some of the table_name.col<n> = cte.col<n>, if the column isn't part of the primary key.
I would always load into a temporary load table first, which doesn't have any unique or PK constraint on those columns. This way you can always see that the whole file has loaded, which is an invaluable check in any ETL work, and for any other easy analysis of the source data.
After that then use an insert such as suggested by an earlier answer, or if you know that the target table is empty then simply
INSERT INTO db_name.dbo.table_name(col1,col2,col3,col4)
SELECT distinct col1,col2,col3,col4 from load_table
The best approach is to use a temporary table and execute a MERGE-INSERT statement. You can do something like this (not tested):
CREATE TABLE #MyTempTable (col1 VARCHAR(50), col2, col3...);
INSERT INTO #MyTempTable(col1, col2, col3, col4)
VALUES (?,?,?,?)
CREATE CLUSTERED INDEX ix_tempCol1 ON #MyTempTable (col1);
MERGE INTO db_name.dbo.table_name AS TARGET
USING #MyTempTable AS SOURCE ON TARGET.COL1 = SOURCE.COL1 AND TARGET.COL2 = SOURCE.COL2 ...
WHEN NOT MATCHED THEN
INSERT(col1, col2, col3, col4)
VALUES(source.col1, source.col2, source.col3, source.col4);
You need to consider the best indexes for your temporary table to make the MERGE faster. With the statement WHEN NOT MATCHED you avoid duplicates depending on the ON clause.
SQL Server Integration Services offers one method that can read data from a source (via a Dataflow task), then remove duplicates using it's Sort control (a checkbox to remove duplicates).
https://www.mssqltips.com/sqlservertip/3036/removing-duplicates-rows-with-ssis-sort-transformation/
Of course the data has to be sorted and 60 million+ rows isn't going to be fast.
If you want to use pure SQL Server then you need a staging table (without a pk constraint). After importing your data into Staging, you would insert into your target table using filtering for the composite PK combination. For example,
Insert into dbo.RealTable (KeyCol1, KeyCol2, KeyCol3, Col4)
Select Col1, Col2, Col3, Col4
from dbo.Staging S
where not exists (Select *
from dbo.RealTable RT
where RT.KeyCol1 = S.Col1
AND RT.KeyCol2 = S.Col2
AND RT.KeyCol3 = S.Col3
)
In theory you could also use the set operator EXCEPT since it takes the distinct values from both tables. For example:
INSERT INTO RealTable
SELECT * FROM Staging
EXCEPT
SELECT * FROM RealTable
Would insert distinct rows from Staging into RealTable (that don't already exist in RealTable). This method doesn't take into account the composite PK using different values on multiple rows- so an insert error would indicate different values are being assigned to the same PK composite key in the csv.
I am new to SQL Server and have a problem with an insert statement. I am to convert an old database to a SQL server relational database. I am transferring the old data into new tables. The old records are not complete which is causing problems because the fields in the new tables do not allow null values. So what I am trying to do is in insert n/a in the missing fields and then use the select statement to retrieve the available data from the old table all at the same time so I don't get null value not allowed, but I get the error Only one expression can be specified in the select list when the subquery is not introduced with EXISTS along with the Insert statement has more columns than the values statement.
I sure there is a way to do this but I can't figure it out, hope someone can help. Below is an abbreviated description to the statement.
insert into database1.dbo.table (col1, col2, .....col10)
values('n/a','n/a',(select col3, col4...col10 from database2.dbo.table)
You can try to use INSERT INTO ... SELECT
INSERT INTO database1.dbo.table (col1, col2, .....col10)
SELECT 'n/a',
'n/a',
col3,
col4,
...col10
FROM database2.dbo.table
I have a Constraint on a table with IGNORE_DUP_KEY. This allows bulk inserts to partially work where some records are dupes and some are not (only inserting the non-dupes). However, it does not allow updates to partially work, where I only want those records updated where dupes will not be created.
Does anyone know how I can support IGNORE_DUP_KEY when applying updates?
I am using MS SQL 2005
If I understand correctly, you want to do UPDATEs without specifying the necessary WHERE logic to avoid creating duplicates?
create table #t (col1 int not null, col2 int not null, primary key (col1, col2))
insert into #t
select 1, 1 union all
select 1, 2 union all
select 2, 3
-- you want to do just this...
update #t set col2 = 1
-- ... but you really need to do this
update #t set col2 = 1
where not exists (
select * from #t t2
where #t.col1 = t2.col1 and col2 = 1
)
The main options that come to mind are:
Use a complete UPDATE statement to avoid creating duplicates
Use an INSTEAD OF UPDATE trigger to 'intercept' the UPDATE and only do UPDATEs that won't create a duplicate
Use a row-by-row processing technique such as cursors and wrap each UPDATE in TRY...CATCH... or whatever the language's equivalent is
I don't think anyone can tell you which one is best, because it depends on what you're trying to do and what environment you're working in. But because row-by-row processing could potentially produce some false positives, I would try to stick with a set-based approach.
I'm not sure what is really going on, but if you are inserting duplicates and updating Primary Keys as part of a bulk load process, then a staging table might be the solution for you. You create a table that you make sure is empty prior to the bulk load, then load it with the 100% raw data from the file, then process that data into your real tables (set based is best). You can do things like this to insert all rows that don't already exist:
INSERT INTO RealTable
(pk, col1, col2, col3)
SELECT
pk, col1, col2, col3
FROM StageTable s
WHERE NOT EXISTS (SELECT
1
FROM RealTable r
WHERE s.pk=r.pk
)
Prevent the duplicates in the first place is best. You could also do UPDATEs on your real table by joining in the staging table, etc. This will avoid the need to "work around" the constraints. When you work around the constraints, you usually create difficult to find bugs.
I have the feeling you should use the MERGE statement and then in the update part you should really not update the key you want to have unique. That also means that you have to define in your table that a key is unique (Setting a unique index or define as primary key). Then any update or insert with a duplicate key will fail.
Edit: I think this link will help on that:
http://msdn.microsoft.com/en-us/library/bb522522.aspx
I tried to make the title as clear as possible... here is my scenario:
I have 2 tables (let's call them table A and table B) that have a similar schema. I would like write a stored procedure that would select specific columns of data out of table A, and insert this data as a new record in table B.
Can someone point me in the write direction to make such a query? I am unsure how to "Hold" the values from the first query, so that I may then perform the insert.
I am trying to avoid making a query, processing it with C# and then making another query...
Thanks.
INSERT INTO B (Col1, Col2) SELECT Col1, Col2 FROM A
Is this what you mean?
You can do this as a single query from C# like this:
Insert into tableB (col1, col2, col3) select col1, col2, col3 from tableA where ...
The trick is that column names need to be in the same order and compatible types.
use a SELECT INTO
SELECT
[Col1],
[COl2]
INTO TableA
FROM TableB