Check for duplicates in an insertion of a stored procedure - sql

Question

I am trying to write a stored procedure that inserts data, but with some fairly simple checks that seem like good practice.
The table currently has 300 columns, of which there is a sequential primary_key_id, a column that we want to check before inserting, say address, a child_of column used when there is new data (what we are inserting) and then the remaining 297 columns.
So let's say the table currently looks like this:
----------------------------------------------------------------------
|PK |Address |child_of |other_attr_1|other_attr2|...
----------------------------------------------------------------------
|1 | 123 Main St |NULL |... |... |...
|2 | 234 South Rd |NULL |... |... |...
|3 | 345 West Rd |NULL |... |... |...
----------------------------------------------------------------------
and we want to add this row, where the address has a new attribute new in the other_attr_1 column. We would use the child_of to reference the primary_key_id of the previous row record. This will allow for a basic history (I hope).
|4 | 123 Main St |1 |new |... |...
How do I check for the duplication in the stored procedure? Do I iterate over each entering parameter with what is already in the DB if it is there?
Here is the code I have thus far:
USE [databaseINeed]
-- SET some_stuff ON --or off :)
-- ....
-- GO
CREATE Procedure [dbo].[insertNonDuplicatedData]
#address text, #other_attr_1 numeric = NULL, #other_attr_2 numeric = NULL, #other_attr_3 numeric = NULL,....;
AS
BEGIN TRY
-- If the address already exists, lets check for updated data
IF EXISTS (SELECT 1 FROM tableName WHERE address = #address)
BEGIN
-- Look at the incoming data vs the data already in the record
--HERE IS WHERE I THINK THE CODE SHOULD GO, WITH SOMETHING LIKE the following pseudocode:
if any attribute parameter values is different than what is already stored
then Insert into tableName (address, child_of, attrs) Values (#address, THE_PRIMARY_KEY_OF_THE_RECORD_THAT_SHARES_THE_ADDRESS, #other_attrs...)
RETURN
END
-- We don't have any data like this, so lets create a new record altogther
ELSE
BEGIN
-- Every time a SQL statement is executed it returns the number of rows that were affected. By using "SET NOCOUNT ON" within your stored procedure you can shut off these messages and reduce some of the traffic.
SET NOCOUNT ON
INSERT INTO tableName (address, other_attr_1, other_attr_2, other_attr_3, ...)
VALUES(#address,#other_attr_1,#other_attr_2,#other_attr_3,...)
END
END TRY
BEGIN CATCH
...
END CATCH
I tried adding a CONSTRAINT on the table itself for all of the 297 attributes that need to be unique when checking against the address column via:
ALTER TABLE tableName ADD CONSTRAINT
uniqueAddressAttributes UNIQUE -- tried also with NONCLUSTERED
(other_attr_1,other_attr_2,...)
but I get an error
ERROR: cannot use more than 32 columns in an index SQL state: 54011
and I think I might be heading down the wrong path trying to rely on the unique constraint.

Answer 1

Surely having such numbers of columns is not a good practice, anyway you can try using a INTERSECT to check the values at once
-- I assume you get the last id to set the
-- THE_PRIMARY_KEY_OF_THE_RECORD_THAT_SHARES_THE_ADDRESS
DECLARE #PK int = (SELECT MAX(PK) FROM tableName WHERE address = #address)
-- No need for an EXISTS(), just check the #PK
IF #PK IS NOT NULL
BEGIN
IF EXISTS(
-- List of attributes from table
-- Possibly very poor performance to get the row by ntext
SELECT other_attr_1, other_attr_2 ... FROM tableName WHERE PK = #PK
INTERSECT
-- List of attributes from variables
SELECT #other_attr_1, #other_attr_2 ...
)
BEGIN
Insert into tableName (address, child_of, attrs) Values
(#address, #PK, #other_attr_1, #other_attr_2 ...)
END
END

Answer 2

With that many columns you could consider doing a hash of all your columns at time of insert, then storing the result in (yet another) column. In your stored procedure you could do the same hash to the input parameters, then check for hash collisions instead of doing field by field comparison on all those fields.
You'd have to probably do some data conversion to make your 300ish columns all nvarchar so they could be concatenated for input into the HASHBYTES function. Also, if any of the columns may be NULL, you'd have to consider how to treat them. For example, if an existing record has field 216 set to NULL and the row attempting to be added is exactly the same, except field 216 is an empty string, is that a match?
Also, with that many columns, the concatenation may run over the max input size of the hashbytes function, so you may need to break it up into multiple hashes of smaller chunks.
That all said, does your architecture really require this 300ish column structure? If you could get away from that, I wouldn't be having to get quite so creative here.

Answer 3

I don't have enough rep to comment, so I am posting as an answer instead.
Eric's SQL should be changed from IF EXISTS to IF NOT EXISTS
I believe the desired logic should be:
If there is an existing address record, check if any attributes are different.
If any attributes are different, insert a new address record, storing the primary key of the latest existing address record in the child_of column
Refactoring Chris & Eric's SQL:
USE [databaseINeed]
-- SET some_stuff ON --or off :)
-- ....
-- GO
CREATE Procedure [dbo].[insertNonDuplicatedData]
#address text, #other_attr_1 numeric = NULL, #other_attr_2 numeric = NULL, #other_attr_3 numeric = NULL,....;
AS
BEGIN TRY
-- If the address already exists, lets check for updated data
IF EXISTS (SELECT 1 FROM tableName WHERE address = #address)
BEGIN
-- Look at the incoming data vs the data already in the record
--HERE IS WHERE I THINK THE CODE SHOULD GO, WITH SOMETHING LIKE the following pseudocode:
DECLARE #PK int = (SELECT MAX(PK) FROM tableName WHERE address = #address)
IF NOT EXISTS(
-- List of attributes from table
-- Possibly very poor performance to get the row by ntext
SELECT other_attr_1, other_attr_2 ... FROM tableName WHERE PK = #PK
INTERSECT
-- List of attributes from variables
SELECT #other_attr_1, #other_attr_2 ...
)
BEGIN
-- #simplyink: existing address record has different combination of (297 column) attribute values
-- at least one attribute column is different (no intersection)
Insert into tableName (address, child_of, attrs) Values
(#address, #PK, #other_attr_1, #other_attr_2 ...)
END
RETURN
END
-- We don't have any data like this, so lets create a new record altogther
ELSE
BEGIN
-- Every time a SQL statement is executed it returns the number of rows that were affected. By using "SET NOCOUNT ON" within your stored procedure you can shut off these messages and reduce some of the traffic.
SET NOCOUNT ON
INSERT INTO tableName (address, other_attr_1, other_attr_2, other_attr_3, ...)
VALUES(#address,#other_attr_1,#other_attr_2,#other_attr_3,...)
END
END TRY
BEGIN CATCH
...
END CATCH