I'm using Merge Replication. The Identity range management is AUTOMATIC
I HAVE A TRIGGER ON COMPANIES TABLE WHICH INSERTS ROWS IN SERIALNUMBERSCHEME TABLE which has documentID as identity column
While synchronizing I'm getting below error
A row insert at 'SERVER\MUMBAI.PROD_SUB' could not be propagated to 'SERVER\NEWYORK.PROD'. This failure can be caused by a constraint violation. Explicit value must be specified for identity column in table 'SerialNumberScheme' either when IDENTITY_INSERT is set to ON or when a replication user is inserting into a NOT FOR REPLICATION identity column.
Data is inserted properly at subscriber but not replicated at publisher
Any solution/suggesstion?
Sounds like your trigger gets fired when the replication agent applies the updates. Normally the trigger should run only at the publisher (or more precisely, at the site which inserts the original data). Then replication will replicate the effect of the trigger. I think that all you need is to mark the trigger as NOT FOR REPLICATION.
See Controlling Constraints, Identities, and Triggers with NOT FOR REPLICATION.
Related
I have a table on premise that is about 21 million rows with a primary key constraint and when I search that table, there are no duplicates. This table is in an OLTP application database that is constantly moving.
I have the exact same table in Azure which has the same primary key constraint. This table is not an application table, it's just a copy of the one that is on-premise (the goal is to use this one for ad hoc queries, as a source for other systems, etc.).
When I use Azure Data Factory to select all_columns from table on premise to the table in Azure, it returns a violation of the primary key constraint. No matter how many times I run this data factory pipeline, it comes back with a primary key violation for duplicate keys (the keys are always changing though).
So I dropped the primary key constraint in Azure and ran the pipeline again, and sure enough, duplication exists.
Upon investigation, it appears that the on-premise database is doing an insert new record then update the old record to inactivate it. So for a fraction of a second, there are two active rows that ADF is grabbing to then try to insert into the table in Azure which of course fails because of duplicate primary keys.
Now to the best of my knowledge, this shouldn't be possible. You can't insert a new row that violates the primary key constraint. But ADF seems to be grabbing all the data and some of those rows are mid-flight where the insert has happened and the update to inactivate the old row hasn't happened yet.
For those that are curious, the insert happens and the update of the old row happens within less than a second... it's typically 10-20 microseconds. I don't know how this is possible and I don't know how to fix it (because I can't modify the application code). The database for the on-premise database is a SQL Server 2000 database and Azure SQL is an Azure SQL database.
Try with readpast hint. It should not select any rows in locking state.
SELECT * FROM yourtable WITH (readpast)
Since you have create_date and updated_date column then you can select rows older than 5 seconds to avoid duplication.
select * from yourtable where created_date<=dateadd(second,-5,getdate()) and updated_date<=dateadd(second,-5,getdate());
Need to enable the Fault tolerance in a Pipeline Azure Data Factory
Copy data from a Source SQL to a Sink SQL database. A primary key is defined in the sink SQL database, but no such primary key is defined in the source SQL server. The duplicated rows that exist in the source cannot be copied to the sink. Copy activity copies only the first row of the source data into the sink. The subsequent source rows that contain the duplicated primary key value are detected as incompatible and are skipped.
To configure Json Definition skip the incompatible rows in copy activity "enableSkipIncompatibleRow": true
Please Refer: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance
If possible to modify your application, need to check the Primary key constraint before insert or update using EXISTS() function.
Example:
IF EXISTS(SELECT * FROM Table_Name WHERE primary key condition)
BEGIN
UPDATE Table_Name
SET Col_Name= value
WHERE condition
END
ELSE
BEGIN
INSERT INTO Table_Name ( col_Name1,col_Name2,,.. )
VALUES ( ‘’,’’,’’,….)
END
Normally when we reinitialise transactional replication it drops the table on the subscriber and recreates it. I want to have a clustered PK on the source database with a non-clustered PK of the same column on the destination and a different clustered index. I understand I can achieve this by temporarily stopping the replication making the changes and enabling it again.
I'm more worried about the future if we ever need to reinitialise I don't want the table to be dropped and lose our different index strategy. I might be being blind but I can't find a setting to allow the table structure on the subscriber to be kept on reinitialisation.
I found the answer. You set this in the article properties of the replication. You can choose to truncate all data if the table already exists.
I'm trying to understand the ramifications of database replication (SQL Server or Golden Gate) for situations where the source database is completely repopulated every night. To clarify, all existing tables are dropped and then the database is reloaded with new tables using same name along with all the data.
Based on my understanding i.e. that replication uses a transaction log... I would assume it will also repeat the process of dropping the tables instead of identifying the differences and just adding the new data. Is that correct?
You can set up the replication using OracleGoldenGate so that it will be doing what you want it to do.
the TRUNCATE TABLE command can be replicated or it can be ignored
the populating of the source table (INSERT/bulk operations) can be replicated or it can be ignored
if a row already exists (meaning a row with the same PK exists) on the target and you INSERT it on the source you can either UPDATE the target or DELETE the old one and INSERT the new, or ignore it
Database replication is based on the redo (transaction) log. Only particular events that appear on the source databases, which are logged can be replicated. But the whole replication engine can make some additional transformations as it is replicating the changes.
i have an table which contains an single column, which is the Primary key, uniqueidentified, rowguid = true. if creating an merge replication, the Agent Fails to start with:
"The article includes only the rowguidcol column. You must publish at least one other column."
is there any way to publish this table, without removing the rowguid and adding an second column?
thx
What is happening is the Snapshot Agent is misidentifying your rowguid column as the ROWGUIDCOL required for each table published in Merge Replication.
See the section Snapshot Considerations in Enhance Merge Replication Performance for more information on the column Merge Replication creates and uses to uniquely identify each row in a published article.
Due to the nature of your existing column, will most likely need to add a second dummy column to get this working for Merge Replication.
I have never seen this before, the rows will be sequential but I have noticed that it skipped over a particular "ID".... 1 2 3 4 6 7 8... missing 5...
There are no transactions in the INSERT stored procedure so nothing to roll back
We do not allow the deletion of records.
What else can be the case?
Probably a failed insert due to some other unique constraint on the table or a foreign key reference in the table and you try to insert an invalid fk value.
The insert doesn't have to be in a transaction.
The identity value increments even on a failed insert.
Igor mentions an important point about identities and transactions. From the docs:
Failed statements and transactions can
change the current identity for a
table and create gaps in the identity
column values. The identity value is
never rolled back even though the
transaction that tried to insert the
value into the table is not committed.
For example, if an INSERT statement
fails because of an IGNORE_DUP_KEY
violation, the current identity value
for the table is still incremented.
Any way, identity counter does not restore your value (if you have executed with transaction or without it). The same behavior has oracle (sequences).
Identity is not transactional.
You may use your own primary key counter and control access to it.
Failed attempts generates an identity value even though it is not inserted into the data table. Then, this results to the lost of an identity (it's a shame I can no longer find the post where I have learned it!).