I will preface this question by saying my sql is not very good :)
We are attempting to build a double buffered implementation in sql by having 2 redundant tables. At any one time 1 of the tables are active and the other inactive. The tables are accessed by a view which we toggle after reloading data. When we reload we want to perform operations on the inactive table and then toggle the active cache on completion.
We have lots of code that looks a bit like:
IF #activeCache = 0
BEGIN WORK ON TABLE 1 AS IT IS INACTIVE
ELSE
BEGIN WORK ON TABLE 0 AS IT IS INACTIVE
The problem is the code between the brackets is not trivial so we end up duplicating the code where the only difference is the table being operated on. We thought table parameters might help but you can't insert into them (which we need to do). My only idea now is to use T4 templates to generate the crud for us but I can't seem to get them to work in a database project.
Is there any sql constructs that can help us? We'd rather not use dynamic sql if possible.
You can use CREATE SYNONYM to effectively create a permanent alias (or "synonym", if you will) for an object. You can run your logic once to decide which table to target, and then run:
CREATE SYNONYM WorkingTable FOR Table1
CREATE SYNONYM MainTable FOR Table2
Then to switch:
DROP SYNONYM WorkingTable
DROP SYNONYM MainTable
CREATE SYNONYM WorkingTable FOR Table2
CREATE SYNONYM MainTable FOR Table1
And everywhere else in your scripts, you can just reference WorkingTable for updates, and MainTable for reads.
That being said, I agree with other comments/answers that question whether this is the best way of working.
Instead of using two tables and switching the view, use a main table and a staging table.
When you are ready to migrate the data to the main table, you can do it in an atomic transaction like so.
begin try
begin tran
delete * from MainTable with (tablockx)
insert MainTable
select * from StagingTable with (tablockx)
commit
end try
begin catch
rollback
raiserror('An error occurred swapping staging data', 16,16)
end catch
That way you always work on the staging table, so there is no difficulty identifying the correct table to use.
Depending on the data you may want to do an incremental update of the main table:
-- delete rows which no longer exist
delete MainTable
from MainTable
where not exists (select 1 from StagingTable where StagingTable.primaryKey = MainTable.primaryKey)
-- Insert new rows
insert MainTable
select *
from StagingTable
where not exists (select 1 from MainTable where StagingTable.primaryKey = MainTable.primaryKey)
-- update rows which have changed
update MainTable
set
col1 = stagingTable.col1,
col2 = stagingTable.col2
from MainTable inner join StagingTable on StagingTable.primaryKey = MainTable.primaryKey
where 1=2
-- Need to compare every column, only update if one is different
-- Both null counts as the same - compare nullity
OR case when MainTable.col1 is null then 0 else 1 end <> case when StagingTable.col1 is null then 0 else 1 end
OR MainTable.col1 <> StagingTable.col1
OR case when MainTable.col2 is null then 0 else 1 end <> case when StagingTable.col2 is null then 0 else 1 end
OR MainTable.col2 <> StagingTable.col2
You can create Inline Table-Valued function which takes #activeCache as a parameter. Select all data from proper table according to parameter in that function. TI'm not sure about performance.
Related
I have had a look at similar problems, however none of the answers helped in my case.
Just a little bit of background. I have Two databases, both have the same table with the same fields and structure. Data already exists in both tables. I want to overwrite and add to the data in db1.table from db2.table the primary ID is causing a problem with the update.
When I use the query:
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT table.field_id,table.field1,table.field2
FROM table;
It works to a blank table, because none of the primary keys exist. As soon as the primary key exists it fails.
Would it be easier for me to overwrite the primary keys? or find the primary key and update the fields related to the field_id? Im really not sure how to go ahead from here. The data needs to be migrated every 5min, so possibly a stored procedure is required?
first you should try to add new records then update all records.you can create a procedure like below code
PROCEDURE sync_Data(a IN NUMBER ) IS
BEGIN
insert into db2.table
select *
from db1.table t
where t.field_id not in (select tt.field_id from db2.table tt);
begin
for t in (select * from db1.table) loop
update db2.table aa
set aa.field1 = t.field1,
aa.field2 = t.field2
where aa.field_id = t.field_id;
end loop;
end;
END sync_Data
Set IsIdentity to No in Identity Specification on the table in which you want to move data, and after executing your script, set it to Yes again
I ended up just removing the data in the new database and sending it again.
DELETE FROM db2.table WHERE db2.table.field_id != 0;
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT table.field_id,table.field1,table.field2
FROM table;
Its not very efficient, but gets the job done. I couldnt figure out the syntax to correctly do an UPDATE or to change the IsIdentity field within MariaDB, so im not sure if they would work or not.
The overhead of deleting and replacing non-trivial amounts of data for an entire table will be prohibitive. That said I'd prefer to update in place (merge) over delete /replace.
USE db1;
INSERT INTO db2.table(field_id,field1,field2)
SELECT t.field_id,t.field1,t.field2
FROM table t
ON DUPLICATE KEY UPDATE field1 = t.field1, field2 = t.field2
This can be used inside a procedure and called every 5 minutes (not recommended) or you could build a trigger that fires on INSERT and UPDATE to keep the tables in sync.
INSERT INTO database1.tabledata SELECT * FROM database2.tabledata;
But you have to keep length of varchar length larger or equal to database2 and keep the same column name
I need to do the following update query through a stored procedure:
UPDATE table1
SET name = #name (this is the stored procedure inputparameter)
WHERE name IS NULL
Table1 has no indexes or keys, 5 columns which are 4 integers and 1 varchar (updatable column 'name' is the varchar column)
The NULL records are about 15.000.000 rows that need updating. This takes about 50 minutes, which I think is too long.
I'm running an Azure SQL DB Standard S6 (400DTU's).
Can anyone give me an advise to improve performance?
As you don't have any keys, or indexes, I can suggest following approach.
1- Create a new table using INTO (which will copy the data) like following query.
SELECT
CASE
WHEN NAME IS NULL THEN #name
ELSE NAME
END AS NAME,
<other columns >
INTO dbo.newtable
FROM table1
2- Drop the old table
drop table table1
3- Rename the new table to table1
exec sp_rename 'dbo.newtable', 'table1'
Another approach can be using batch update, sometime you get better performance compared to bulk update (You need to test by adjusting the batch size).
WHILE EXISTS (SELECT 1 FROM table1 WHERE name is null)
BEGIN
UPDATE TOP (10000) table1
SET name = #name
WHERE n ame is null
END
can you do with following method ?
UPDATE table1
SET name = ISNULL(name,#name)
for null values it will update with #name and rest will be updated with same value.
No. You are updating 15,000,000 rows which is going to take a long time. Each update has overhead for finding the row and logging the value.
With so many rows to update, it is unlikely that the overhead is finding the rows. If you add an index on name, the update is going to actually have to update the index as well as updating the original values.
If your concern is locking the database, you can set up a loop where you do something like this over and over:
UPDATE TOP (100000) table1
SET name = #name (this is the stored procedure inputparameter)
WHERE name IS NULL;
100,000 rows should be about 30 seconds or so.
In this case, an index on name does help. Otherwise, each iteration of the loop would in essence be reading the entire table.
I want to track the update changes in a table via a trigger:
CREATE TABLE dbo.TrackTable(...columns same as target table)
GO
CREATE TRIGGER dboTrackTable
ON dbo.TargetTable
AFTER UPDATE
AS
INSERT INTO dbo.TrackTable (...columns)
SELECT (...columns)
FROM Inserted
However in real production some of the update queries select rows with vague conditions and update them all regardless of whether they are actually changed, like
UPDATE Targettable
SET customer_type = 'VIP'
WHERE 1 = 1
--or is_obsolete = 0 or register_date < '20160101' something
But due to table size and to analyze, I only want to choose those actually modified data for tracking. How to achieve this goal?
My track table has many columns (so I do not prefer checking inserted and deleted column one by one) but it seldom changes structure.
I guess the following code will be useful.
CREATE TABLE dbo.TrackTable(...columns same as target table)
GO
CREATE TRIGGER dboTrackTable
ON dbo.TargetTable
AFTER UPDATE
AS
INSERT INTO dbo.TrackTable (...columns)
SELECT *
FROM Inserted
EXCEPT
SELECT *
FROM Deleted
I realize this post is a couple months old now, but for anyone looking for a well-rounded answer:
To exit the trigger if no rows were affected on SQL Server 2016 and up, Microsoft recommends using the built-in ROWCOUNT_BIG() function in the Optimizing DML Triggers section of the Create Trigger documentation.
Usage:
IF ROWCOUNT_BIG() = 0
RETURN;
To ensure you are excluding rows that were not changed, you'll need to do a compare of the inserted and deleted tables inside the trigger. Taking your example code:
INSERT INTO dbo.TrackTable (...columns)
SELECT (...columns)
FROM Inserted i
INNER JOIN deleted d
ON d.[SomePrimaryKeyCol]=i.[SomePrimaryKeyCol] AND
i.customer_type<>d.customer_type
Microsoft documentation and w3schools are great resources for learning how to leverage various types of queries and trigger best practices.
Prevent trigger from doing anything if no rows changed.
Writing-triggers-the-right-way
CREATE TRIGGER the_trigger on dbo.Data
after update
as
begin
if ##ROWCOUNT = 0
return
set nocount on
/* Some Code Here */
end
Get a list of rows that changed:
CREATE TRIGGER the_trigger on dbo.data
AFTER UPDATE
AS
SELECT * from inserted
Previous stack overflow on triggers
#anna - as per #Oded's answer, when an update is performed, the rows are in the deleted table with the old information, and the inserted table with the new information –
I have resolved this problem because I have overlooked something that is already part of my code and this situation is not needed.
In SQL Server 2008, I have two IF statements
If value = ''
begin
select * into #temptable from table 1
end
Else If value <> ''
begin
select * into #temptable from table 2
end
but when I try to execute it gives me because of the second
temptable:
There is already an object named '#temptable' in the database.
I don't want to use another temp table name as I would have to change the after code a lot. Is there a way to bypass this?
I would recommend making some changes so that your code is a little more maintainable. One problem with the way you have it set up here is with the SELECT * syntax you're using. If you later decide to make a change to the schema of table1 or table2, you could have non-obvious consequences. In production code, it's better to spell these things out so that it's clear exactly which columns you're using and where.
Also, are you really using all of the columns from table 1 and table 2 in the code that follows? You might be taking a performance hit loading more data than you need. I'd go through the code that uses #temptable and figure out which columns it's actually using. Then start by creating your temp table:
CREATE TABLE #temptable(col1 int, col2 int, col3 int, col4 int)
Include all of the possible columns that could be used, even if some of them might be null in certain cases. Presumably, the code that follows already understands that. Then you can set up your IF statements:
IF value = ''
BEGIN
INSERT INTO #temptable(col1, col2, col3)
SELECT x,y,z
FROM table1
END
ELSE
INSERT INTO #temptable(col1, col4)
SELECT alpha,beta
FROM table2
END
Your SELECT statement, as written, is creating the temp table and INSERTING into it all in one statement. Create the temp table separately with a CREATE TABLE statement, then INSERT INTO in your two IF statements.
Using SELECT INTO creates the table on the fly, as you know. Even if your query only referenced #temptable once, if you were to run it more than once (without dropping the table after the first run), you would get the same error (although if it were inside a stored procedure, it would probably only exist in the scope of the stored procedure).
However, you can't even compile this query. Using the Parse command (Ctrl+F5) on the following query, for example, fails even though the same table is used as the source table.
select * into #temptable from SourceTable
select * into #temptable from SourceTable
If the structure of tables 1 and 2 were the same, you could do something like the following.
select * into #temptable from
(select * from Table1 where #value = ''
union
select * from Table2 where #value <> '') as T
If, however, the tables have different structures, then I'm not sure what you can do, other than what agt and D. Lambert recommended.
If I have a table with this fields:
int:id_account
int:session
string:password
Now for a login statement I run this sql UPDATE command:
UPDATE tbl_name
SET session = session + 1
WHERE id_account = 17 AND password = 'apple'
Then I check if a row was affected, and if one indeed was affected I know that the password was correct.
Next what I want to do is retrieve all the info of this affected row so I'll have the rest of the fields info.
I can use a simple SELECT statement but I'm sure I'm missing something here, there must be a neater way you gurus know, and going to tell me about (:
Besides it bothered me since the first login sql statement I ever written.
Is there any performance-wise way to combine a SELECT into an UPDATE if the UPDATE did update a row?
Or am I better leaving it simple with two statements? Atomicity isn't needed, so I might better stay away from table locks for example, no?
You should use the same WHERE statement for SELECT. It will return the modified rows, because your UPDATE did not change any columns used for lookup:
UPDATE tbl_name
SET session = session + 1
WHERE id_account = 17 AND password = 'apple';
SELECT *
FROM tbl_name
WHERE id_account = 17 AND password = 'apple';
An advice: never store passwords as plain text! Use a hash function, like this:
MD5('apple')
There is ROW_COUNT() (do read about details in the docs).
Following up by SQL is ok and simple (which is always good), but it might unnecessary stress the system.
This won't work for statements such as...
Update Table
Set Value = 'Something Else'
Where Value is Null
Select Value From Table
Where Value is Null
You would have changed the value with the update and would be unable to recover the affected records unless you stored them beforehand.
Select * Into #TempTable
From Table
Where Value is Null
Update Table
Set Value = 'Something Else'
Where Value is Null
Select Value, UniqueValue
From #TempTable TT
Join Table T
TT.UniqueValue = T.UniqueValue
If you're lucky, you may be able to join the temp table's records to a unique field within Table to verify the update. This is just one small example of why it is important to enumerate records.
You can get the effected rows by just using ##RowCount..
select top (Select ##RowCount) * from YourTable order by 1 desc