We have a big database (100 GB) in our production environment and we have two mirrors. we use replication to sync our mirrored database.(Transactional Replication on SQL Server 2012)
when we created replication, our DBA first download full backup from our servers and restore it on Mirror server ,then configured the replication between Publisher (Production-DB) and Subscriber (Mirror-1). Everything works fine and mirror and production are synced perfectly.
before each release, we should test our system on our UAT server with REAL! data. so we create a back up from one of our Mirrored Servers (onsite mirror) and Restore it on our UAT Database.
All table in our database have one PK like :
[ID] [int] IDENTITY(1,1) NOT NULL
In our UAT Database the last record of OrderTransaction table ID = 160.000.000, but when our application try to insert into this table, system encounter a problem and exception raised.
Violation of PRIMARY KEY constraint ‘ID’. Cannot insert duplicate key in object ...
system try to insert an integer value smaller than 160.000.000 (ID = 145.695.000)
as i queried the database this id is the last ID when the backup of production db restored to mirrored db and after that transactional replication started.
all information on production and mirrored are synced correctly and ID of the Tables on both servers are the same and identical.
When I run this command on the UAT or Mirrored Database it returns 145.695.000
SELECT IDENT_CURRENT( 'OrderTransaction' )
as I read MSDN, blogs and related SO question, this problem related to replication and identity columns.
But, how can i force our Mirror database to grow its Identity with Production.(and accept this value as local identity value). Or may be we should write a script that reseed every table for testing in our UAT environment!?
thanks.
If you are only having issues when restoring a prod mirror into UAT your simplest solution, as you suggest, is likely to just script a reseed for each table.
For more general problems replication and identity problem you may find this excellent article from Hilary Cotter on Simple-Talk useful.
In this article we looked at ways to implement identity management
solutions in replication to avoid primary key conflicts. We looked at
partitioning and manual and automatic identity range management
solutions. Solutions using automatic identity range management can work well with careful planning.
EDIT:
This should reset all the identities to the current max value:
EXEC sp_MSForEachTable '
IF OBJECTPROPERTY(object_id(''?''), ''TableHasIdentity'') = 1
DBCC CHECKIDENT (''?'', RESEED)'
Which is a slight modification to Pinal Dave's work.
Related
In one of my tables Fee in column "ReceiptNo" in SQL Server 2012 database identity increment suddenly started jumping to 100s instead of 1 depending on the following two things.
if it is 1205446 it is jumps to 1206306, if it is 1206321, it jumps to 1207306 and if it is 1207314, it jumps to 1208306. What I want to make you note is that the last three digits remain constant i.e 306 whenever the jumping occurs as shown in the following picture.
this problem occurs when I restart my computer
You are encountering this behaviour due to a performance improvement since SQL Server 2012.
It now by default uses a cache size of 1,000 when allocating IDENTITY values for an int column and restarting the service can "lose" unused values (The cache size is 10,000 for bigint/numeric).
This is mentioned in the documentation
SQL Server might cache identity values for performance reasons and
some of the assigned values can be lost during a database failure or
server restart. This can result in gaps in the identity value upon
insert. If gaps are not acceptable then the application should use its
own mechanism to generate key values. Using a sequence generator with
the NOCACHE option can limit the gaps to transactions that are never
committed.
From the data you have shown it looks like this happened after the data entry for 22 December then when it restarted SQL Server reserved the values 1206306 - 1207305. After data entry for 24 - 25 December was done another restart and SQL Server reserved the next range 1207306 - 1208305 visible in the entries for the 28th.
Unless you are restarting the service with unusual frequency any "lost" values are unlikely to make any significant dent in the range of values allowed by the datatype so the best policy is not to worry about it.
If this is for some reason a real issue for you some possible workarounds are...
You can use a SEQUENCE instead of an identity column and define a smaller cache size for example and use NEXT VALUE FOR in a column default.
Or apply trace flag 272 which makes the IDENTITY allocation logged as in versions up to 2008 R2. This applies globally to all databases.
Or, for recent versions, execute ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE = OFF to disable the identity caching for a specific database.
You should be aware none of these workarounds assure no gaps. This has never been guaranteed by IDENTITY as it would only be possible by serializing inserts to the table. If you need a gapless column you will need to use a different solution than either IDENTITY or SEQUENCE
This problems occurs after restarting the SQL Server.
The solution is:
Run SQL Server Configuration Manager.
Select SQL Server Services.
Right-click SQL Server and select Properties.
In the opening window under Startup Parameters, type -T272 and click Add, then press Apply button and restart.
From SQL Server 2017+ you could use ALTER DATABASE SCOPED CONFIGURATION:
IDENTITY_CACHE = { ON | OFF }
Enables or disables identity cache at the database level. The default
is ON. Identity caching is used to improve INSERT performance on
tables with Identity columns. To avoid gaps in the values of the
Identity column in cases where the server restarts unexpectedly or
fails over to a secondary server, disable the IDENTITY_CACHE option.
This option is similar to the existing SQL Server Trace Flag 272,
except that it can be set at the database level rather than only at
the server level.
(...)
G. Set IDENTITY_CACHE
This example disables the identity cache.
ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE=OFF ;
I know my answer might be late to the party. But i have solved in another way by adding a start up stored procedure in SQL Server 2012.
Create a following stored procedure in master DB.
USE [master]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[ResetTableNameIdentityAfterRestart]
AS
BEGIN
begin TRAN
declare #id int = 0
SELECT #id = MAX(id) FROM [DatabaseName].dbo.[TableName]
--print #id
DBCC CHECKIDENT ('[DatabaseName].dbo.[TableName]', reseed, #id)
Commit
END
Then add it in to Start up by using following syntax.
EXEC sp_procoption 'ResetOrderIdentityAfterRestart', 'startup', 'on';
This is a good idea if you have few tables. but if you have to do for many tables, this method still works but not a good idea.
This is still a very common issue among many developers and applications regardless of size.
Unfortunately the suggestions above do not fix all scenarios, i.e. Shared hosting, you cannot rely on your host to set the -t272 startup parameter.
Also, if you have existing tables that use these identity columns for primary keys, it is a HUGE effort to drop those columns and recreate new ones to use the BS sequence workaround. The Sequence workaround is only good if you are designing the tables new from scratch in SQL 2012+
Bottom line is, if you are on Sql Server 2008R2, then STAY ON IT. Seriously, stay on it. Until Microsoft admits that they introduced a HUGE bug, which is still there even in Sql Server 2016, then we should not upgrade until they own it and FIX IT.
Microsoft straight up introduced a breaking change, i.e. they broke a working API that no longer works as designed, due to the fact that their system forgets their current identity on a restart. Cache or no cache, this is unacceptable, and the Microsoft developer by the name of Bryan needs to own it, instead of tell the world that it is "by design" and a "feature". Sure, the caching is a feature, but losing track of what the next identity should be, IS NOT A FEATURE. It's a fricken BUG!!!
I will share the workaround that I used, because My DB's are on Shared Hosting servers, also, I am not dropping and recreating my Primary Key columns, that would be a huge PITA.
Instead, this is my shameful hack (but not as shameful as this POS bug that microsoft has introduced).
Hack/Fix:
Before your insert commands, just reseed your identity before each insert. This fix is only recommended if you don't have admin control over your Sql Server instance, otherwise I suggest reseeding on restart of server.
declare #newId int -- where int is the datatype of your PKey or Id column
select #newId = max(YourBuggedIdColumn) from YOUR_TABLE_NAME
DBCC CheckIdent('YOUR_TABLE_NAME', RESEED, #newId)
Just those 3 lines immediately before your insert, and you should be good to go. It really won't affect performance that much, i.e. it will be unnoticeable.
Goodluck.
There are many possible reasons for jumping identity values. They range from rolled back inserts to identity management for replication. What is causing this in your case I can't tell without spending some time in your system.
You should know however, that in no case you can assume an identity column to be contiguos. There are just too many things that can cause gaps.
You can find a little more information about this here: http://sqlity.net/en/792/the-gap-in-the-identity-value-sequence/
In one of my tables Fee in column "ReceiptNo" in SQL Server 2012 database identity increment suddenly started jumping to 100s instead of 1 depending on the following two things.
if it is 1205446 it is jumps to 1206306, if it is 1206321, it jumps to 1207306 and if it is 1207314, it jumps to 1208306. What I want to make you note is that the last three digits remain constant i.e 306 whenever the jumping occurs as shown in the following picture.
this problem occurs when I restart my computer
You are encountering this behaviour due to a performance improvement since SQL Server 2012.
It now by default uses a cache size of 1,000 when allocating IDENTITY values for an int column and restarting the service can "lose" unused values (The cache size is 10,000 for bigint/numeric).
This is mentioned in the documentation
SQL Server might cache identity values for performance reasons and
some of the assigned values can be lost during a database failure or
server restart. This can result in gaps in the identity value upon
insert. If gaps are not acceptable then the application should use its
own mechanism to generate key values. Using a sequence generator with
the NOCACHE option can limit the gaps to transactions that are never
committed.
From the data you have shown it looks like this happened after the data entry for 22 December then when it restarted SQL Server reserved the values 1206306 - 1207305. After data entry for 24 - 25 December was done another restart and SQL Server reserved the next range 1207306 - 1208305 visible in the entries for the 28th.
Unless you are restarting the service with unusual frequency any "lost" values are unlikely to make any significant dent in the range of values allowed by the datatype so the best policy is not to worry about it.
If this is for some reason a real issue for you some possible workarounds are...
You can use a SEQUENCE instead of an identity column and define a smaller cache size for example and use NEXT VALUE FOR in a column default.
Or apply trace flag 272 which makes the IDENTITY allocation logged as in versions up to 2008 R2. This applies globally to all databases.
Or, for recent versions, execute ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE = OFF to disable the identity caching for a specific database.
You should be aware none of these workarounds assure no gaps. This has never been guaranteed by IDENTITY as it would only be possible by serializing inserts to the table. If you need a gapless column you will need to use a different solution than either IDENTITY or SEQUENCE
This problems occurs after restarting the SQL Server.
The solution is:
Run SQL Server Configuration Manager.
Select SQL Server Services.
Right-click SQL Server and select Properties.
In the opening window under Startup Parameters, type -T272 and click Add, then press Apply button and restart.
From SQL Server 2017+ you could use ALTER DATABASE SCOPED CONFIGURATION:
IDENTITY_CACHE = { ON | OFF }
Enables or disables identity cache at the database level. The default
is ON. Identity caching is used to improve INSERT performance on
tables with Identity columns. To avoid gaps in the values of the
Identity column in cases where the server restarts unexpectedly or
fails over to a secondary server, disable the IDENTITY_CACHE option.
This option is similar to the existing SQL Server Trace Flag 272,
except that it can be set at the database level rather than only at
the server level.
(...)
G. Set IDENTITY_CACHE
This example disables the identity cache.
ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE=OFF ;
I know my answer might be late to the party. But i have solved in another way by adding a start up stored procedure in SQL Server 2012.
Create a following stored procedure in master DB.
USE [master]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[ResetTableNameIdentityAfterRestart]
AS
BEGIN
begin TRAN
declare #id int = 0
SELECT #id = MAX(id) FROM [DatabaseName].dbo.[TableName]
--print #id
DBCC CHECKIDENT ('[DatabaseName].dbo.[TableName]', reseed, #id)
Commit
END
Then add it in to Start up by using following syntax.
EXEC sp_procoption 'ResetOrderIdentityAfterRestart', 'startup', 'on';
This is a good idea if you have few tables. but if you have to do for many tables, this method still works but not a good idea.
This is still a very common issue among many developers and applications regardless of size.
Unfortunately the suggestions above do not fix all scenarios, i.e. Shared hosting, you cannot rely on your host to set the -t272 startup parameter.
Also, if you have existing tables that use these identity columns for primary keys, it is a HUGE effort to drop those columns and recreate new ones to use the BS sequence workaround. The Sequence workaround is only good if you are designing the tables new from scratch in SQL 2012+
Bottom line is, if you are on Sql Server 2008R2, then STAY ON IT. Seriously, stay on it. Until Microsoft admits that they introduced a HUGE bug, which is still there even in Sql Server 2016, then we should not upgrade until they own it and FIX IT.
Microsoft straight up introduced a breaking change, i.e. they broke a working API that no longer works as designed, due to the fact that their system forgets their current identity on a restart. Cache or no cache, this is unacceptable, and the Microsoft developer by the name of Bryan needs to own it, instead of tell the world that it is "by design" and a "feature". Sure, the caching is a feature, but losing track of what the next identity should be, IS NOT A FEATURE. It's a fricken BUG!!!
I will share the workaround that I used, because My DB's are on Shared Hosting servers, also, I am not dropping and recreating my Primary Key columns, that would be a huge PITA.
Instead, this is my shameful hack (but not as shameful as this POS bug that microsoft has introduced).
Hack/Fix:
Before your insert commands, just reseed your identity before each insert. This fix is only recommended if you don't have admin control over your Sql Server instance, otherwise I suggest reseeding on restart of server.
declare #newId int -- where int is the datatype of your PKey or Id column
select #newId = max(YourBuggedIdColumn) from YOUR_TABLE_NAME
DBCC CheckIdent('YOUR_TABLE_NAME', RESEED, #newId)
Just those 3 lines immediately before your insert, and you should be good to go. It really won't affect performance that much, i.e. it will be unnoticeable.
Goodluck.
There are many possible reasons for jumping identity values. They range from rolled back inserts to identity management for replication. What is causing this in your case I can't tell without spending some time in your system.
You should know however, that in no case you can assume an identity column to be contiguos. There are just too many things that can cause gaps.
You can find a little more information about this here: http://sqlity.net/en/792/the-gap-in-the-identity-value-sequence/
In one of my tables Fee in column "ReceiptNo" in SQL Server 2012 database identity increment suddenly started jumping to 100s instead of 1 depending on the following two things.
if it is 1205446 it is jumps to 1206306, if it is 1206321, it jumps to 1207306 and if it is 1207314, it jumps to 1208306. What I want to make you note is that the last three digits remain constant i.e 306 whenever the jumping occurs as shown in the following picture.
this problem occurs when I restart my computer
You are encountering this behaviour due to a performance improvement since SQL Server 2012.
It now by default uses a cache size of 1,000 when allocating IDENTITY values for an int column and restarting the service can "lose" unused values (The cache size is 10,000 for bigint/numeric).
This is mentioned in the documentation
SQL Server might cache identity values for performance reasons and
some of the assigned values can be lost during a database failure or
server restart. This can result in gaps in the identity value upon
insert. If gaps are not acceptable then the application should use its
own mechanism to generate key values. Using a sequence generator with
the NOCACHE option can limit the gaps to transactions that are never
committed.
From the data you have shown it looks like this happened after the data entry for 22 December then when it restarted SQL Server reserved the values 1206306 - 1207305. After data entry for 24 - 25 December was done another restart and SQL Server reserved the next range 1207306 - 1208305 visible in the entries for the 28th.
Unless you are restarting the service with unusual frequency any "lost" values are unlikely to make any significant dent in the range of values allowed by the datatype so the best policy is not to worry about it.
If this is for some reason a real issue for you some possible workarounds are...
You can use a SEQUENCE instead of an identity column and define a smaller cache size for example and use NEXT VALUE FOR in a column default.
Or apply trace flag 272 which makes the IDENTITY allocation logged as in versions up to 2008 R2. This applies globally to all databases.
Or, for recent versions, execute ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE = OFF to disable the identity caching for a specific database.
You should be aware none of these workarounds assure no gaps. This has never been guaranteed by IDENTITY as it would only be possible by serializing inserts to the table. If you need a gapless column you will need to use a different solution than either IDENTITY or SEQUENCE
This problems occurs after restarting the SQL Server.
The solution is:
Run SQL Server Configuration Manager.
Select SQL Server Services.
Right-click SQL Server and select Properties.
In the opening window under Startup Parameters, type -T272 and click Add, then press Apply button and restart.
From SQL Server 2017+ you could use ALTER DATABASE SCOPED CONFIGURATION:
IDENTITY_CACHE = { ON | OFF }
Enables or disables identity cache at the database level. The default
is ON. Identity caching is used to improve INSERT performance on
tables with Identity columns. To avoid gaps in the values of the
Identity column in cases where the server restarts unexpectedly or
fails over to a secondary server, disable the IDENTITY_CACHE option.
This option is similar to the existing SQL Server Trace Flag 272,
except that it can be set at the database level rather than only at
the server level.
(...)
G. Set IDENTITY_CACHE
This example disables the identity cache.
ALTER DATABASE SCOPED CONFIGURATION SET IDENTITY_CACHE=OFF ;
I know my answer might be late to the party. But i have solved in another way by adding a start up stored procedure in SQL Server 2012.
Create a following stored procedure in master DB.
USE [master]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER PROCEDURE [dbo].[ResetTableNameIdentityAfterRestart]
AS
BEGIN
begin TRAN
declare #id int = 0
SELECT #id = MAX(id) FROM [DatabaseName].dbo.[TableName]
--print #id
DBCC CHECKIDENT ('[DatabaseName].dbo.[TableName]', reseed, #id)
Commit
END
Then add it in to Start up by using following syntax.
EXEC sp_procoption 'ResetOrderIdentityAfterRestart', 'startup', 'on';
This is a good idea if you have few tables. but if you have to do for many tables, this method still works but not a good idea.
This is still a very common issue among many developers and applications regardless of size.
Unfortunately the suggestions above do not fix all scenarios, i.e. Shared hosting, you cannot rely on your host to set the -t272 startup parameter.
Also, if you have existing tables that use these identity columns for primary keys, it is a HUGE effort to drop those columns and recreate new ones to use the BS sequence workaround. The Sequence workaround is only good if you are designing the tables new from scratch in SQL 2012+
Bottom line is, if you are on Sql Server 2008R2, then STAY ON IT. Seriously, stay on it. Until Microsoft admits that they introduced a HUGE bug, which is still there even in Sql Server 2016, then we should not upgrade until they own it and FIX IT.
Microsoft straight up introduced a breaking change, i.e. they broke a working API that no longer works as designed, due to the fact that their system forgets their current identity on a restart. Cache or no cache, this is unacceptable, and the Microsoft developer by the name of Bryan needs to own it, instead of tell the world that it is "by design" and a "feature". Sure, the caching is a feature, but losing track of what the next identity should be, IS NOT A FEATURE. It's a fricken BUG!!!
I will share the workaround that I used, because My DB's are on Shared Hosting servers, also, I am not dropping and recreating my Primary Key columns, that would be a huge PITA.
Instead, this is my shameful hack (but not as shameful as this POS bug that microsoft has introduced).
Hack/Fix:
Before your insert commands, just reseed your identity before each insert. This fix is only recommended if you don't have admin control over your Sql Server instance, otherwise I suggest reseeding on restart of server.
declare #newId int -- where int is the datatype of your PKey or Id column
select #newId = max(YourBuggedIdColumn) from YOUR_TABLE_NAME
DBCC CheckIdent('YOUR_TABLE_NAME', RESEED, #newId)
Just those 3 lines immediately before your insert, and you should be good to go. It really won't affect performance that much, i.e. it will be unnoticeable.
Goodluck.
There are many possible reasons for jumping identity values. They range from rolled back inserts to identity management for replication. What is causing this in your case I can't tell without spending some time in your system.
You should know however, that in no case you can assume an identity column to be contiguos. There are just too many things that can cause gaps.
You can find a little more information about this here: http://sqlity.net/en/792/the-gap-in-the-identity-value-sequence/
To fine tune the performance of overall system, I was checking the existing table's Indexes and found that we are using a ErrorLog table which is hit(for writing warnings and errors) for millions of transactions everyday. As we have indexes(on datetime) on this kind of table I thought this logging will definitely takes longer than the logging in to table without any indexes.
The whole indexing on CreateDateTime is used only by Developers for querying the table for troubleshooting in Production environment. Is it possible to take out index on the primary production server and have index only for the table in secondary(backup) db server. As we are doing replication on secondary server data is always in Sysc.
To Sync both the tables via replication, do we need to have same indexes on both tables?
Assuming transactional replication, there is nothing saying that the indexes have to be the same between the publisher and the subscriber. The only thing that you need to keep is the primary key as that's how replication identifies what row(s) on the subscriber need to be affected by a given statement at the publisher.
I have a SQL Server 2005 Database which uses merge replication.
I have added four new stored procedures on the publisher end, but they haven't replicated.
The publication property "Subscription Options" - "Replicate schema changes" is set to True.
How can I get easily get these tables replicated without having to send new snapshots to all of the subscribers?
I found this bit of code, perhaps it will help.
create table table2(pk int not null primary key, charcol char(20))
GO
sp_addarticle 'paul','table2','table2'
GO
sp_refreshsubscriptions 'paul'
GO
sp_addsubscription 'paul','table2',##servername,'paulsub'
GO
Adding the article and then running the snapshot agent seems to reinit the entire publication.