In sql how to always insert data in a table with concurrency? Must ensure that the data is received, example table "Bet", all the app clients of that database server must have de ensure that their bet is placed.
On an INSERT statement, the only way to not have concurrency is to violate some kind of constraint. As long as you have declared the constraint (primary key, not null, foreign key, etc), the database will throw an error on any violation.
I'm not sure what API you are using to talk to the database, but it will certainly signal in some way that a database error has occurred. Then you need to handle that case appropriately by informing the application to invalidate the data.
Related
I have a table on premise that is about 21 million rows with a primary key constraint and when I search that table, there are no duplicates. This table is in an OLTP application database that is constantly moving.
I have the exact same table in Azure which has the same primary key constraint. This table is not an application table, it's just a copy of the one that is on-premise (the goal is to use this one for ad hoc queries, as a source for other systems, etc.).
When I use Azure Data Factory to select all_columns from table on premise to the table in Azure, it returns a violation of the primary key constraint. No matter how many times I run this data factory pipeline, it comes back with a primary key violation for duplicate keys (the keys are always changing though).
So I dropped the primary key constraint in Azure and ran the pipeline again, and sure enough, duplication exists.
Upon investigation, it appears that the on-premise database is doing an insert new record then update the old record to inactivate it. So for a fraction of a second, there are two active rows that ADF is grabbing to then try to insert into the table in Azure which of course fails because of duplicate primary keys.
Now to the best of my knowledge, this shouldn't be possible. You can't insert a new row that violates the primary key constraint. But ADF seems to be grabbing all the data and some of those rows are mid-flight where the insert has happened and the update to inactivate the old row hasn't happened yet.
For those that are curious, the insert happens and the update of the old row happens within less than a second... it's typically 10-20 microseconds. I don't know how this is possible and I don't know how to fix it (because I can't modify the application code). The database for the on-premise database is a SQL Server 2000 database and Azure SQL is an Azure SQL database.
Try with readpast hint. It should not select any rows in locking state.
SELECT * FROM yourtable WITH (readpast)
Since you have create_date and updated_date column then you can select rows older than 5 seconds to avoid duplication.
select * from yourtable where created_date<=dateadd(second,-5,getdate()) and updated_date<=dateadd(second,-5,getdate());
Need to enable the Fault tolerance in a Pipeline Azure Data Factory
Copy data from a Source SQL to a Sink SQL database. A primary key is defined in the sink SQL database, but no such primary key is defined in the source SQL server. The duplicated rows that exist in the source cannot be copied to the sink. Copy activity copies only the first row of the source data into the sink. The subsequent source rows that contain the duplicated primary key value are detected as incompatible and are skipped.
To configure Json Definition skip the incompatible rows in copy activity "enableSkipIncompatibleRow": true
Please Refer: https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance
If possible to modify your application, need to check the Primary key constraint before insert or update using EXISTS() function.
Example:
IF EXISTS(SELECT * FROM Table_Name WHERE primary key condition)
BEGIN
UPDATE Table_Name
SET Col_Name= value
WHERE condition
END
ELSE
BEGIN
INSERT INTO Table_Name ( col_Name1,col_Name2,,.. )
VALUES ( ‘’,’’,’’,….)
END
I have a staging table without any constraints in my Azure SQL database (Azure SQL database 12.0.2000.8). I want to insert the data from the Staging table into the "real" table on which multiple constraints are set. When inserting the data, I use a statement of the kind
INSERT INTO <someTable> SELECT <columns> FROM StagingTable;
Now I only get the first error when violating some constraints. However, for my use case, it is important to get all violations, so they can be resolved altogether.
I have tried using TRY...CATCH mechanisms, however, this will throw an error on the first error and run the catch clause, but it will not continue with the other data. Note that the correct data that has no violations should not be inserted, so the whole insert statement can be rolled back on one error, however, I want to see all violations to be able to correct them all without having to run the insert statement multiple times to get all errors.
EDIT:
The types of constraints that need to be checked are foreign key constraints, NOT NULL constraints, duplicate keys. No casting is done, so no need to check for conversions.
There are couple of options:
If you want to catch row level information, you have to go for cursors or while loop and try to insert each row in TRY CATCH block and see if you are getting any error, and log the same.
Create another table similar to main table(say, MainCheckTable) with all constraints and disable all the constraints and load the data.
Now, you can leverage DBCC CHECKCONSTRAINTS to see all the constraint violations.Read more on this .
USE DBName;
DBCC CHECKCONSTRAINTS(MainCheckTable) WITH ALL_CONSTRAINTS;
First, don't look at your primary table(s). Look at the related tables e.g. lookups etc. Populate these first. Once you have populated the related tables (i.e.) satisfy all related constraints, then add the data.
You need to work backwards from the least constrained tables to the most constrained if that makes sense.
You should check that your related tables have the required reference values/fields that you intend to insert. This is easy to do, since you already have a staging table.
I am building a .NET disconnected client-server application that uses Entity Framework 5 (EF5) to generate a SQL Server CE 4.0 database from POCOs. The application allows the user to perform a bulk copy of data from the network SQL Server into the client's SQL Server CE database. This is very (VERY) slow, due to the constraints and indexes created by EF5. Temporarily dropping the constraints and indexes will reduce the 30-minute wait to 1 minute or less.
Before starting the bulk copy, the application executes queries to drop the constraints and indexes from the SQL Server CE tables. However, the commands fail, because EF5 created constraint names include the table schema name, dot, and table name. The dot in the constraint name is causing the drop command to fail, due to a parsing issue.
For example, POCO Customer creates table dbo.Customer with the primary key constraint PK_dbo.Customer_Id. The database performs as expected.
However, upon executing non-query:
ALTER TABLE Customer DROP CONSTRAINT PK_dbo.Customer;
SQL Server Compact ADO.NET Data Provider returns an error:
There was an error parsing the query.
[ Token line number = 1, Token line offset = 57, Token in error = . ]
Of course, using a secondary DataContext object that does not have foreign keys generate the database without the constraints, and then add them later works; but, that requires maintaining two DataContext objects and hopefully not forgetting to keep both updated. Therefore, I am looking for one of two solutions:
Compose the DROP statement in such a way that the . character is parsed
Prevent EF5 from using the . character in the constraint and index names
Thank you in advance for your help!
Wrap that bad boy in a []. It tells the parser that everything inside is the key name.
ALTER TABLE Customer DROP CONSTRAINT [PK_dbo.Customer];
Should run fine.
Personally I just wrap every identifier in brackets to avoid this exact issue. So I would write this query like this.
ALTER TABLE [Customer] DROP CONSTRAINT [PK_dbo.Customer];
I think it's more readable that way because you can instantly see identifiers.
I have two tables in two different databases. In table1 (in database1) there is a column called column1 and it is a primary key. Now in table2 (in database2) there is a column called column2 and I want to add it as a foreign key.
I tried to add it and it gave me the following error:
Msg 1763, Level 16, State 0, Line 1
Cross-database foreign key references are not supported. Foreign key Database2.table2.
Msg 1750, Level 16, State 0, Line 1
Could not create constraint. See previous errors.
How do I do that since the tables are in different databases.
You would need to manage the referential constraint across databases using a Trigger.
Basically you create an insert, update trigger to verify the existence of the Key in the Primary key table. If the key does not exist then revert the insert or update and then handle the exception.
Example:
Create Trigger dbo.MyTableTrigger ON dbo.MyTable, After Insert, Update
As
Begin
If NOT Exists(select PK from OtherDB.dbo.TableName where PK in (Select FK from inserted) BEGIN
-- Handle the Referential Error Here
END
END
Edited: Just to clarify. This is not the best approach with enforcing referential integrity. Ideally you would want both tables in the same db but if that is not possible. Then the above is a potential work around for you.
If you need rock solid integrity, have both tables in one database, and use an FK constraint. If your parent table is in another database, nothing prevents anyone from restoring that parent database from an old backup, and then you have orphans.
This is why FK between databases is not supported.
You could use check constraint with a user defined function to make the check. It is more reliable than a trigger. It can be disabled and reenabled when necessary same as foreign keys and rechecked after a database2 restore.
CREATE FUNCTION dbo.fn_db2_schema2_tb_A
(#column1 INT)
RETURNS BIT
AS
BEGIN
DECLARE #exists bit = 0
IF EXISTS (
SELECT TOP 1 1 FROM DB2.SCHEMA2.tb_A
WHERE COLUMN_KEY_1 = #COLUMN1
) BEGIN
SET #exists = 1
END;
RETURN #exists
END
GO
ALTER TABLE db1.schema1.tb_S
ADD CONSTRAINT CHK_S_key_col1_in_db2_schema2_tb_A
CHECK(dbo.fn_db2_schema2_tb_A(key_col1) = 1)
In my experience, the best way to handle this when the primary authoritative source of information for two tables which are related has to be in two separate databases is to sync a copy of the table from the primary location to the secondary location (using T-SQL or SSIS with appropriate error checking - you cannot truncate and repopulate a table while it has a foreign key reference, so there are a few ways to skin the cat on the table updating).
Then add a traditional FK relationship in the second location to the table which is effectively a read-only copy.
You can use a trigger or scheduled job in the primary location to keep the copy updated.
The short answer is that SQL Server (as of SQL 2008) does not support cross database foreign keys--as the error message states.
While you cannot have declarative referential integrity (the FK), you can reach the same goal using triggers. It's a bit less reliable, because the logic you write may have bugs, but it will get you there just the same.
See the SQL docs # http://msdn.microsoft.com/en-us/library/aa258254%28v=sql.80%29.aspx Which state:
Triggers are often used for enforcing
business rules and data integrity. SQL
Server provides declarative
referential integrity (DRI) through
the table creation statements (ALTER
TABLE and CREATE TABLE); however, DRI
does not provide cross-database
referential integrity. To enforce
referential integrity (rules about the
relationships between the primary and
foreign keys of tables), use primary
and foreign key constraints (the
PRIMARY KEY and FOREIGN KEY keywords
of ALTER TABLE and CREATE TABLE). If
constraints exist on the trigger
table, they are checked after the
INSTEAD OF trigger execution and prior
to the AFTER trigger execution. If the
constraints are violated, the INSTEAD
OF trigger actions are rolled back and
the AFTER trigger is not executed
(fired).
There is also an OK discussion over at SQLTeam - http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=31135
Achieving referential integrity accross databases is not an easy task.
Here is a list of frequently employed mechanisms:
Clone & Sync: The referenced data is regularly cloned/merged into the referencing database. This may be suitable if the referenced data is rarely changing. You end up with two physical copies of the same data, and need a reliable process to keep them in sync (e.g. with an ETL pipeline).
Triggers: Changes to the referencing data and the referenced data are caught by SQL triggers, which ensure referential integrity. However, triggers can be slow, and may not fire at a database restore. It cannot hurt to run scheduled consistency checks as part of the operations monitoring. Write access to the referenced database is required for installing and maintaining the trigger.
Check constraints: SQL-Server offers user-defined contraints, which ensure that every row satisfies a given condition. One can exploit this functionality by writing a user defined function that checks the existence of a row in the referenced data, and then use this function as a CHECK's predicate in the referencing table. This does not catch changes in the referenced data. It is an RDBMS-specific solution, but works accross server boundaries (e.g. using linked servers). It is a good choice for referencing globally unique IDs, such as article codes in a company's ERP system, which never get deleted or re-assigned.
Re-think database architecture: When all the above mechanisms are unsatisfactory, multiple databases may be merged in a single database. The originating database names can become schema names, allowing effective grouping of database objects.
As the error message says, this is not supported on sql server.
The only way to ensure refrerential integrity is to work with triggers.
I have two tables in two different databases. In table1 (in database1) there is a column called column1 and it is a primary key. Now in table2 (in database2) there is a column called column2 and I want to add it as a foreign key.
I tried to add it and it gave me the following error:
Msg 1763, Level 16, State 0, Line 1
Cross-database foreign key references are not supported. Foreign key Database2.table2.
Msg 1750, Level 16, State 0, Line 1
Could not create constraint. See previous errors.
How do I do that since the tables are in different databases.
You would need to manage the referential constraint across databases using a Trigger.
Basically you create an insert, update trigger to verify the existence of the Key in the Primary key table. If the key does not exist then revert the insert or update and then handle the exception.
Example:
Create Trigger dbo.MyTableTrigger ON dbo.MyTable, After Insert, Update
As
Begin
If NOT Exists(select PK from OtherDB.dbo.TableName where PK in (Select FK from inserted) BEGIN
-- Handle the Referential Error Here
END
END
Edited: Just to clarify. This is not the best approach with enforcing referential integrity. Ideally you would want both tables in the same db but if that is not possible. Then the above is a potential work around for you.
If you need rock solid integrity, have both tables in one database, and use an FK constraint. If your parent table is in another database, nothing prevents anyone from restoring that parent database from an old backup, and then you have orphans.
This is why FK between databases is not supported.
You could use check constraint with a user defined function to make the check. It is more reliable than a trigger. It can be disabled and reenabled when necessary same as foreign keys and rechecked after a database2 restore.
CREATE FUNCTION dbo.fn_db2_schema2_tb_A
(#column1 INT)
RETURNS BIT
AS
BEGIN
DECLARE #exists bit = 0
IF EXISTS (
SELECT TOP 1 1 FROM DB2.SCHEMA2.tb_A
WHERE COLUMN_KEY_1 = #COLUMN1
) BEGIN
SET #exists = 1
END;
RETURN #exists
END
GO
ALTER TABLE db1.schema1.tb_S
ADD CONSTRAINT CHK_S_key_col1_in_db2_schema2_tb_A
CHECK(dbo.fn_db2_schema2_tb_A(key_col1) = 1)
In my experience, the best way to handle this when the primary authoritative source of information for two tables which are related has to be in two separate databases is to sync a copy of the table from the primary location to the secondary location (using T-SQL or SSIS with appropriate error checking - you cannot truncate and repopulate a table while it has a foreign key reference, so there are a few ways to skin the cat on the table updating).
Then add a traditional FK relationship in the second location to the table which is effectively a read-only copy.
You can use a trigger or scheduled job in the primary location to keep the copy updated.
The short answer is that SQL Server (as of SQL 2008) does not support cross database foreign keys--as the error message states.
While you cannot have declarative referential integrity (the FK), you can reach the same goal using triggers. It's a bit less reliable, because the logic you write may have bugs, but it will get you there just the same.
See the SQL docs # http://msdn.microsoft.com/en-us/library/aa258254%28v=sql.80%29.aspx Which state:
Triggers are often used for enforcing
business rules and data integrity. SQL
Server provides declarative
referential integrity (DRI) through
the table creation statements (ALTER
TABLE and CREATE TABLE); however, DRI
does not provide cross-database
referential integrity. To enforce
referential integrity (rules about the
relationships between the primary and
foreign keys of tables), use primary
and foreign key constraints (the
PRIMARY KEY and FOREIGN KEY keywords
of ALTER TABLE and CREATE TABLE). If
constraints exist on the trigger
table, they are checked after the
INSTEAD OF trigger execution and prior
to the AFTER trigger execution. If the
constraints are violated, the INSTEAD
OF trigger actions are rolled back and
the AFTER trigger is not executed
(fired).
There is also an OK discussion over at SQLTeam - http://www.sqlteam.com/forums/topic.asp?TOPIC_ID=31135
Achieving referential integrity accross databases is not an easy task.
Here is a list of frequently employed mechanisms:
Clone & Sync: The referenced data is regularly cloned/merged into the referencing database. This may be suitable if the referenced data is rarely changing. You end up with two physical copies of the same data, and need a reliable process to keep them in sync (e.g. with an ETL pipeline).
Triggers: Changes to the referencing data and the referenced data are caught by SQL triggers, which ensure referential integrity. However, triggers can be slow, and may not fire at a database restore. It cannot hurt to run scheduled consistency checks as part of the operations monitoring. Write access to the referenced database is required for installing and maintaining the trigger.
Check constraints: SQL-Server offers user-defined contraints, which ensure that every row satisfies a given condition. One can exploit this functionality by writing a user defined function that checks the existence of a row in the referenced data, and then use this function as a CHECK's predicate in the referencing table. This does not catch changes in the referenced data. It is an RDBMS-specific solution, but works accross server boundaries (e.g. using linked servers). It is a good choice for referencing globally unique IDs, such as article codes in a company's ERP system, which never get deleted or re-assigned.
Re-think database architecture: When all the above mechanisms are unsatisfactory, multiple databases may be merged in a single database. The originating database names can become schema names, allowing effective grouping of database objects.
As the error message says, this is not supported on sql server.
The only way to ensure refrerential integrity is to work with triggers.