Why is the SQL updating some rows when no changes are detected? - sql

Question

I have a statement I use to add or update a row of data. I'm observing the SQL updating a row when no apparent changes are present. I'm trying to work out why it's insisting on updating the row when the SQL should prevent it.
I'm updating about 1,000 rows this way (one after another) and only 2 rows seem to update when the mask I've created suggests they shouldn't.
Here's the relevant SQL snippet:
...
ELSE IF NOT EXISTS (
SELECT PERMIT_STATUS, PERMIT_LOCATION, PERMIT_COMMODITY, PERMIT_TYPE_CODE, PERMIT_TYPE_DESCRIPTION, PERMIT_ALLOCATION_METHOD, PERMIT_OPERATION_NAME, OWNERS, SHARE_PERCENTAGE, OPERATOR, MINERALS, PERMIT_DURATION_YEARS, PERMIT_DURATION_MONTHS, PERMIT_AREA, PERMIT_AREA_UNIT, PERMIT_OFFSHORE_ONSHORE, PERMIT_STATUS_DATE, PERMIT_MINERAL_GROUP, PERMIT_MP, SUBSEQUENT_TO_PERMIT, PERMIT_COMMENCEMENT_DATE, PERMIT_EXPIRY_DATE, PERMIT_GRANT_DATE, PERMIT_NONEXCLUSIVE_YN, ShapeGeoJson, Removed
FROM MyTable
WHERE PERMIT_NUMBER = #PERMIT_NUMBER
INTERSECT
SELECT #PERMIT_STATUS, #PERMIT_LOCATION, #PERMIT_COMMODITY, #PERMIT_TYPE_CODE, #PERMIT_TYPE_DESCRIPTION, #PERMIT_ALLOCATION_METHOD, #PERMIT_OPERATION_NAME, #OWNERS, #SHARE_PERCENTAGE, #OPERATOR, #MINERALS, #PERMIT_DURATION_YEARS, #PERMIT_DURATION_MONTHS, #PERMIT_AREA, #PERMIT_AREA_UNIT, #PERMIT_OFFSHORE_ONSHORE, #PERMIT_STATUS_DATE, #PERMIT_MINERAL_GROUP, #PERMIT_MP, #SUBSEQUENT_TO_PERMIT, #PERMIT_COMMENCEMENT_DATE, #PERMIT_EXPIRY_DATE, #PERMIT_GRANT_DATE, #PERMIT_NONEXCLUSIVE_YN, #ShapeGeoJson, #Removed
)
BEGIN
UPDATE MyTable
SET PERMIT_STATUS = #PERMIT_STATUS
,PERMIT_LOCATION = #PERMIT_LOCATION
,PERMIT_COMMODITY = #PERMIT_COMMODITY
,PERMIT_TYPE_CODE = #PERMIT_TYPE_CODE
,PERMIT_TYPE_DESCRIPTION = #PERMIT_TYPE_DESCRIPTION
,PERMIT_ALLOCATION_METHOD = #PERMIT_ALLOCATION_METHOD
,PERMIT_OPERATION_NAME = #PERMIT_OPERATION_NAME
,OWNERS = #OWNERS
,SHARE_PERCENTAGE = #SHARE_PERCENTAGE
,OPERATOR = #OPERATOR
,MINERALS = #MINERALS
,PERMIT_DURATION_YEARS = #PERMIT_DURATION_YEARS
,PERMIT_DURATION_MONTHS = #PERMIT_DURATION_MONTHS
,PERMIT_AREA = #PERMIT_AREA
,PERMIT_AREA_UNIT = #PERMIT_AREA_UNIT
,PERMIT_OFFSHORE_ONSHORE = #PERMIT_OFFSHORE_ONSHORE
,PERMIT_STATUS_DATE = #PERMIT_STATUS_DATE
,PERMIT_MINERAL_GROUP = #PERMIT_MINERAL_GROUP
,PERMIT_MP = #PERMIT_MP
,SUBSEQUENT_TO_PERMIT = #SUBSEQUENT_TO_PERMIT
,PERMIT_COMMENCEMENT_DATE = #PERMIT_COMMENCEMENT_DATE
,PERMIT_EXPIRY_DATE = #PERMIT_EXPIRY_DATE
,PERMIT_GRANT_DATE = #PERMIT_GRANT_DATE
,PERMIT_NONEXCLUSIVE_YN = #PERMIT_NONEXCLUSIVE_YN
,ShapeGeoJson = #ShapeGeoJson
,UpdatedUtc = GETUTCDATE()
,Removed = #Removed
,UpdatedMask = (
IIF ( PERMIT_STATUS != #PERMIT_STATUS, 4096, 0 ) +
IIF ( PERMIT_LOCATION != #PERMIT_LOCATION, 256, 0 ) +
IIF ( PERMIT_COMMODITY != #PERMIT_COMMODITY, 512, 0 ) +
IIF ( PERMIT_TYPE_CODE != #PERMIT_TYPE_CODE, 64, 0 ) +
IIF ( PERMIT_TYPE_DESCRIPTION != #PERMIT_TYPE_DESCRIPTION, 128, 0 ) +
IIF ( PERMIT_ALLOCATION_METHOD != #PERMIT_ALLOCATION_METHOD, 1024, 0 ) +
IIF ( PERMIT_OPERATION_NAME != #PERMIT_OPERATION_NAME, 2048, 0 ) +
IIF ( OWNERS != #OWNERS, 1048576, 0 ) +
IIF ( SHARE_PERCENTAGE != #SHARE_PERCENTAGE, 2097152, 0 ) +
IIF ( OPERATOR != #OPERATOR, 4194304, 0 ) +
IIF ( MINERALS != #MINERALS, 8388608, 0 ) +
IIF ( PERMIT_DURATION_YEARS != #PERMIT_DURATION_YEARS, 32768, 0 ) +
IIF ( PERMIT_DURATION_MONTHS != #PERMIT_DURATION_MONTHS, 65536, 0 ) +
IIF ( PERMIT_AREA != #PERMIT_AREA, 67108864, 0 ) +
IIF ( PERMIT_AREA_UNIT != #PERMIT_AREA_UNIT, 134217728, 0 ) +
IIF ( PERMIT_OFFSHORE_ONSHORE != #PERMIT_OFFSHORE_ONSHORE, 262144, 0 ) +
IIF ( PERMIT_STATUS_DATE != #PERMIT_STATUS_DATE, 8192, 0 ) +
IIF ( PERMIT_MINERAL_GROUP != #PERMIT_MINERAL_GROUP, 131072, 0 ) +
IIF ( PERMIT_MP != #PERMIT_MP, 524288, 0 ) +
IIF ( SUBSEQUENT_TO_PERMIT != #SUBSEQUENT_TO_PERMIT, 33554432, 0 ) +
IIF ( PERMIT_COMMENCEMENT_DATE != #PERMIT_COMMENCEMENT_DATE, 16, 0 ) +
IIF ( PERMIT_EXPIRY_DATE != #PERMIT_EXPIRY_DATE, 32, 0 ) +
IIF ( PERMIT_GRANT_DATE != #PERMIT_GRANT_DATE, 8, 0 ) +
IIF ( PERMIT_NONEXCLUSIVE_YN != #PERMIT_NONEXCLUSIVE_YN, 16384, 0 ) +
IIF ( ShapeGeoJson != #ShapeGeoJson, 137438953472, 0 ) +
IIF ( Removed != #Removed, 274877906944, 0 )
)
WHERE PERMIT_NUMBER = #PERMIT_NUMBER;
END
...
I'm using the INTERSECT method at the top to detect a change in any of the columns of row data. This is because it's easier to compare NULL values in columns.
And you can see the UpdatedMask is being updated with a value to indicate which columns have been updated. When the problem occurs, this is being set to 0 (and when working as expected, I see a correct UpdatedMask).
I can't see any typos or obvious logic errors. So I wonder if it might be due to load on SQL Azure somehow from updating so many records one after another with a sizable ShapeGeoJson column?
Here's the table definition in case it's useful:
CREATE TABLE [dbo].[MyTable](
[PERMIT_NUMBER] [varchar](30) NOT NULL,
[PERMIT_STATUS] [varchar](30) NULL,
[PERMIT_LOCATION] [varchar](4000) NULL,
[PERMIT_COMMODITY] [varchar](8) NULL,
[PERMIT_TYPE_CODE] [varchar](4000) NULL,
[PERMIT_TYPE_DESCRIPTION] [varchar](255) NULL,
[PERMIT_ALLOCATION_METHOD] [varchar](4000) NULL,
[PERMIT_OPERATION_NAME] [varchar](4000) NULL,
[OWNERS] [varchar](4000) NULL,
[SHARE_PERCENTAGE] [varchar](4000) NULL,
[OPERATOR] [varchar](220) NULL,
[MINERALS] [varchar](4000) NULL,
[PERMIT_DURATION_YEARS] [varchar](4000) NULL,
[PERMIT_DURATION_MONTHS] [varchar](4000) NULL,
[PERMIT_AREA] [varchar](4000) NULL,
[PERMIT_AREA_UNIT] [varchar](4) NULL,
[PERMIT_OFFSHORE_ONSHORE] [varchar](4000) NULL,
[PERMIT_STATUS_DATE] [date] NULL,
[PERMIT_MINERAL_GROUP] [varchar](4000) NULL,
[PERMIT_MP] [varchar](4000) NULL,
[SUBSEQUENT_TO_PERMIT] [varchar](4000) NULL,
[PERMIT_COMMENCEMENT_DATE] [date] NULL,
[PERMIT_EXPIRY_DATE] [date] NULL,
[PERMIT_GRANT_DATE] [date] NULL,
[PERMIT_NONEXCLUSIVE_YN] [char](1) NULL,
[ShapeGeoJson] [varchar](max) NOT NULL,
[CreatedUtc] [datetime] NOT NULL,
[UpdatedUtc] [datetime] NOT NULL,
[Removed] [bit] NOT NULL,
[UpdatedMask] [bigint] NULL,
CONSTRAINT [PK_MyTable_1] PRIMARY KEY CLUSTERED
(
[PERMIT_NUMBER] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
Any ideas?
The update process runs nightly so I'll keep my eye on the rows that seem to break my rules to see if there's a pattern there. I've seen this a few times now, but neglected to preserve the data for investigating previously.

Answer 1

In your code you have code duplication. You have the INTERSECT/EXISTS part where you check whether something has been updated or not and then you have the code calculating UpdatedMask where the logic gets repeated.
I would suggest to first remove the duplication of logic. Then it will be much easier to find the problem and fix it on one place.
Below code shows you an alternative way of updating the records and only relying on the code to calculate UpdatedMask.
-- We will update MyTable from data in the NewData alias that contains
-- all of the new values and UpdatedMask
UPDATE U
SET U.PERMIT_STATUS = NewData.PERMIT_STATUS
,U.PERMIT_LOCATION = NewData.PERMIT_LOCATION
.... all other columns ....
,U.UpdatedMask = NewData.UpdatedMask
FROM MyTable U
INNER JOIN (
SELECT T.PERMIT_STATUS = #PERMIT_STATUS
,T.PERMIT_LOCATION = #PERMIT_LOCATION
... all other columns ...
,UpdatedMask = (
IIF ( T.PERMIT_STATUS != #PERMIT_STATUS, 4096, 0 ) +
IIF ( T.PERMIT_LOCATION != #PERMIT_LOCATION, 256, 0 ) +
... all other columns ....
)
FROM MyTable T
WHERE T.PERMIT_NUMBER = #PERMIT_NUMBER
-- In the join clause we only take rows that are actually updated
) AS NewData ON NewData.UpdatedMask<>0
WHERE U.PERMIT_NUMBER = #PERMIT_NUMBER;

Answer 2

I managed to drill down to the issue. The logic for calculating the UpdatedMask value does not take into consideration NULL values when comparing.
So the update is valid. It's the calculated UpdatedMask that's invalid.
So now I've uglified my SQL even further:
UpdatedMask = (
IIF ( ISNULL(PERMIT_STATUS, '') != ISNULL(#PERMIT_STATUS, ''), 4096, 0 ) +
IIF ( ISNULL(PERMIT_LOCATION, '') != ISNULL(#PERMIT_LOCATION, ''), 256, 0 ) +
...
)