My question is a bit tricky, because it's mostly a logical problem.
I've tried to optimize my app speed by reading everything into memory but only those records, which changed since "last read" = greatest timestamp of records last time loaded.
FirebirdSQL database engine does not allow to update a field in an "After Trigger" directly, so it's obviously using "before update or insert" triggers to update the field new.last_changed = current_timestamp;
The problem:
As it turns out, this is a totally WRONG method, because those triggers fire on transaction start!
So if there is a transaction that takes some more time than an other, the saved "last changed time" will be lower than a short-burst transaction fired and finished in between.
1. tr.: 13:00:01.400 .............................Commit << this record will be skipped !
2. tr.: 13:00.01.500......Commit << reading of data will happen here.
The next read will be >= 13:00.01.500
I've tried:
to rewrite all triggers, so they fire after and call an UPDATE orders SET ... << but this causing circular, self-calling endless events.
Would a SET_CONTEXT lock interfere with multi-row update and nested triggers?
(I do not see any possibility this method would work good if running multiple updates in the same transaction.)
What is the common solution for all this?
Edit1:
What I want to happen is to read only those records from DB actually changed since last read. For that to happen, I need the engine to update records AFTER COMMIT. (Not during it, "in the middle".)
This trigger is NOT good, because it will fire on the moment of change, (not after Commit):
alter trigger SYNC_ORDERS active after insert or update position 999 AS
declare variable N timestamp;
begin
N = cast('NOW' as timestamp);
if (new.last_changed <> :N) then
update ORDERS set last_changed= :N where ID=new.ID;
end
And from the application I do:
Query1.SQL.Text := 'SELECT * FROM orders WHERE last_changed >= ' + DateTimeToStr( latest_record );
Query1.Open;
latest_record := Query1.FieldByName('last_changed').asDateTime;
.. this code will list only the record commited in the 2th transaction (earlier) and never the first, longer running transaction (commited later).
Edit2:
It seems I have the same question as here... , but specially for FirebirdSQL.
There are not really any good solutions there, but gave me an idea:
- What if I create an extra table and log changes earlier than 5 minutes there per table?
- Before each SQL query, first I will ask for any changes in that table, sequenced via ID grow!
- Delete lines older than 23 hours
ID TableID Changed
===========================
1 5 2019.11.27 19:36:21
2 5 2019.11.27 19:31:19
Edit3:
As Arioch already suggested, one solution is to:
create a "logger table" filled on every BEFORE INSERT OR UPDATE
trigger by every table
and update the "last_changed" sequence of it
by the ON TRANSACTION COMMIT trigger
But, would not be ...
a better approach?:
adding 1-1 last_sequence INT64 DEFAULT NULL column to every table
create a global generator LAST_GEN
update every table's every NULL row with a gen_id(LAST_GEN,1) inside the ON TRANSACTION COMMIT trigger
SET to NULL again on every BEFORE INSERT OR UPDATE trigger
So basically switching the last_sequence column of a record to:
NULL > 1 > NULL > 34 ... every time it gets modified.
This way I :
do not have to fill the DB with log data,
and I can query the tables directly with WHERE last_sequence>1;.
No needed to pre-query the "logger table" first.
I'm just afraid: WHAT happens, if the ON TRANSACTION COMMIT trigger is trying to update a last_sequence field, while a 2th transaction's ON BEFORE trigger is locking the record (of an other table)?
Can this happen at all?
The final solution is based on the idea, that:
Each table's BEFORE INSERT OR UPDATE trigger can push a time of the transaction: RDB$SET_CONTEXT('USER_TRANSACTION', 'table31', current_timestamp);
The global ON TRANSACTION COMMIT trigger can insert a sequence + time into a "logging table", if receiving such a context.
It can even take care of "daylight saving changes" and "intervals", by logging only "big time differences", like >=1 minute, to reduce the amount of records.)
A stored procedure can ease and speed up the calculation of 'LAST_QUERY_TIME' of each query's.
Example:
1.)
create trigger ORDERS_BI active before insert or update position 0 AS
BEGIN
IF (NEW.ID IS NULL) THEN
NEW.ID = GEN_ID(GEN_ORDERS,1);
RDB$SET_CONTEXT('USER_TRANSACTION', 'orders_table', current_timestamp);
END
2, 3.)
create trigger TRG_SYNC_AFTER_COMMIT ACTIVE ON transaction commit POSITION 1 as
declare variable N TIMESTAMP;
declare variable T VARCHAR(255);
begin
N = cast('NOW' as timestamp);
T = RDB$GET_CONTEXT('USER_TRANSACTION', 'orders_table');
if (:T is not null) then begin
if (:N < :T) then T = :N; --system time changed eg.: daylight saving" -1 hour
if (datediff(second from :T to :N) > 60 ) then --more than 1min. passed
insert into "SYNC_PAST_TIMES" (ID, TABLE_NUMBER, TRG_START, SYNC_TIME, C_USER)
values (GEN_ID(GEN_SYNC_PAST_TIMES, 1), 31, cast(:T as timestamp), :N, CURRENT_USER);
end;
-- other tables too:
T = RDB$GET_CONTEXT('USER_TRANSACTION', 'details_table');
-- ...
when any do EXIT;
end
Edit1:
It is possible to speed up the readout of the "last-time-changed" value from our SYNC_PAST_TIMES table with a help of a Stored Procedure. Logically, You have to store in memory both the ID PT_ID + the time PT_TM in your program to call it for each table.
CREATE PROCEDURE SP_LAST_MODIF_TIME (
TABLE_NUMBER SM_INT,
LAST_PASTTIME_ID BG_INT,
LAST_PASTTIME TIMESTAMP)
RETURNS (
PT_ID BG_INT,
PT_TM TIMESTAMP)
AS
declare variable TEMP_TIME TIMESTAMP;
declare variable TBL SMALLINT;
begin
PT_TM = :LAST_PASTTIME;
FOR SELECT p.ID, p.SYNC_TIME, p.TABLA FROM SYNC_PAST_TIMES p WHERE (p.ID > :LAST_PASTTIME_ID)
ORDER by p.ID ASC
INTO PT_ID, TEMP_TIME, TBL DO --the PT_ID gets an increasing value immediately
begin
if (:TBL = :TABLE_NUMBER) then
if (:TEMP_TIME< :MI_TIME) then
PT_TM = :TEMP_TIME; --searching for the smallest
end
if (:PT_ID IS NULL) then begin
PT_ID = :LAST_PASTTIME_ID;
PT_TM = :LAST_PASTTIME;
end
suspend;
END
You can use this procedure by including in your select, using the WITH .. AS format:
with UTLS as (select first 1 PT_ID, PT_TM from SP_LAST_MODIF_TIME (55, -- TABLE_NUMBER
0, '1899.12.30 00:00:06.000') ) -- last PT_ID, PT_TM from your APP
select first 1000 u.PT_ID, current_timestamp as NOWWW, r.*
from UTLS u, "Orders" r
where (r.SYNC_TIME >= u.PT_TM);
Using FIRST 1000 is a must to prevent reading the whole table if all values are changed at once.
Upgrading the SQL, adding a new column, etc. makes SYNC_TIME changing to NOW at the same time at all rows of the table.
You may adjust it per table individually, just like the interval of seconds to monitor changes. Add a check to your APP, how to handle the case, if the new data reaches 1000 lines at once ...
Related
CREATE OR ALTER TRIGGER CheckQuantity
ON dbo.Products
AFTER UPDATE
AS
BEGIN
UPDATE dbo.Products
SET Product_ReOrder = 1
FROM Inserted i
WHERE i.Product_ID = dbo.Products.Product_ID
AND i.Product_QOH < 5;
I am not getting a syntax error
syntax error near ;
This is referring to the ; at the end of the code.
Not 100% sure what you're trying to do - you're not giving us much to go on, either!
I'm assuming you mean you want to set a column called Product_ReOrder in your table to 1 if another column Product_QOH is less than 5 - correct?
In that case - use a trigger something like this:
CREATE OR ALTER TRIGGER CheckQuantity
ON dbo.Products
AFTER UPDATE
AS
BEGIN
UPDATE dbo.Products
SET Product_ReOrder = 1
FROM Inserted i
WHERE i.PrimaryKeyColumn = dbo.Products.PrimaryKeyColumn
AND i.Product_QOH < 5;
END
The trigger will fire after an UPDATE, and Inserted will contain all rows (can and will be multiple rows!) that have been updated - so I'm assuming you want to check the quantity on those rows.
I'm joining the base table (dbo.Products) to the Inserted pseudo table on the primary key column of your table (which we don't know what it is - so you need to adapt this as needed), and I'm setting the Product_ReOrder column to 1, if the Products_QOH value is less than 5.
Your line of code
Select #QOH = (select Product_QOH from inserted)
has a fatal flaw of assuming that only one row was updated - this might be the case sometimes - but you cannot rely on that! Your trigger must be capable of handling multiple rows being updated - because the trigger is called only once, even if 10 rows are updated with a command - and then Inserted will contain all those 10 updated rows. Doing such a select is dangerous - you'll get one arbitrary row, and you'll ignore all the rest of them ....
Is that what you're looking for?
I'm unclear what you were thinking when you wrote this code, or what template you were basing off, but there are many syntax errors.
It seems you probably want something like this:
The update() function only tells us if that column was present in the update statement, not if anything was actually changed.
We need to check if we are being called recursively, in order to bail out.
We also check if no rows have been changed at all, and bail out early
Note how inserted and deleted are compared to see if any rows actually changed. This also deals correctly with multiple rows.
We then need to rejoin Products in order to update it.
create or alter trigger CheckQuantity
on Products
after update
as
set nocount on;
if not(update(Products_QOH))
or TRIGGER_NESTLEVEL(##PROCID, 'AFTER', 'DML') > 1
or not exists (select 1 from inserted)
return; -- early bail-out
update p
set Product_ReOrder = 1
from Products p
join (
select i.YourPrimaryKey, i.Products_QOH
from inserted i
where i.Product_QOH < 5
except
select d.YourPrimaryKey, d.Products_QOH
from deleted d
) i on i.YourPrimaryKey = p.YourPrimaryKey;
However, I don't understand why you are using a trigger at all.
I strongly suggest you use a computed column for this instead:
ALTER TABLE Products
DROP COLUMN Product_ReOrder;
ALTER TABLE Products
ADD Product_ReOrder AS (CASE WHEN Product_QOH < 5 THEN CAST(1 AS bit) ELSE CAST(0 AS bit) END);
I need to delete a bunch of records (literally millions) but I don't want to make it in an individual statement, because of performance issues. So I created a view:
CREATE VIEW V1
AS
SELECT FIRST 500000 *
FROM TABLE
WHERE W_ID = 14
After that I do a bunch deletes for example:
DELETE FROM V1 WHERE TS < 2021-01-01
What I want is to import this logic in a While loop and in stored procedure. I tried SELECT COUNT query like this:
SELECT COUNT(*)
FROM TABLE
WHERE W_ID = 14 AND TS < 2021-01-01;
Can I use this number in the same procedure as a condition and how can I manage that?
This is what I have tried and I get an error
ERROR: Dynamic SQL Error; SQL error code = -104; Token unknown; WHILE
Code:
CREATE PROCEDURE DeleteBatch
AS
DECLARE VARIABLE CNT INT;
BEGIN
SELECT COUNT(*) FROM TABLE WHERE W_ID = 14 AND TS < 2021-01-01 INTO :cnt;
WHILE cnt > 0 do
BEGIN
IF (cnt > 0) THEN
DELETE FROM V1 WHERE TS < 2021-01-01;
END
ELSE break;
END
I just can't wrap my head around this.
To clarify, in my previous question I wanted to know how to manage the garbage_collection after many deleted records, and I did what was suggested - SELECT * FROM TABLE; or gfix -sweep and that worked very well. As mentioned in the comments the correct statement is SELECT COUNT(*) FROM TABLE;
After that another even bigger database was given to me - above 50 million. And the problem was the DB was very slow to operate with. And I managed to get the server it was on, killed with a DELETE statement to clean the database.
That's why I wanted to try deleting in batches. The slow-down problem there was purely hardware - HDD has gone, and we replaced it. After that there was no problem with executing statements and doing backup and restore to reclaim disk space.
Provided the data that you need to delete, doesn't ever need to be rollbacked once the stored procedure is kicked off, there is another way to handle massive DELETEs in a Stored Procedure.
The example stored procedure will delete the rows 500,000 at a time. It will loop until there aren't any more rows to delete. The AUTONOMOUS TRANSACTION will allow you to put each delete statement in its own transaction and it will commit immediately after the statement completes. This is issuing an implicit commit inside a stored procedure, which you normally can't do.
CREATE OR ALTER PROCEDURE DELETE_TABLEXYZ_ROWS
AS
DECLARE VARIABLE RC INTEGER;
BEGIN
RC = 9999;
WHILE (RC > 0) DO
BEGIN
IN AUTONOMOUS TRANSACTION DO
BEGIN
DELETE FROM TABLEXYZ ROWS 500000;
RC = ROW_COUNT;
END
END
SELECT COUNT(*)
FROM TABLEXYZ
INTO :RC;
END
because of performance issues
What are those exactly? I do not think you actually are improving performance, by just running delete in loops but within the same transaction, or even different TXs but within the same timespan. You seem to be solving some wrong problem. The issue is not how you create "garbage", but how and when Firebird "collects" it.
For example, Select Count(*) in Interbase/Firebird engines means natural scan over all the table and the garbage collection is often trigggered by it, which can itself get long if lot of garbage was created (and massive delete surely does, no matter if done by one million-rows statement or million of one-row statements).
How to delete large data from Firebird SQL database
If you really want to slow down deletion - you have to spread that activity round the clock, and make your client application call a deleting SP for example once every 15 minutes. You would have to add some column to the table, flagging it is marked for deletion and then do the job like that
CREATE PROCEDURE DeleteBatch(CNT INT)
AS
DECLARE ROW_ID INTEGER;
BEGIN
FOR SELECT ID FROM TABLENAME WHERE MARKED_TO_DEL > 0 INTO :row_id
DO BEGIN
CNT = CNT - 1;
DELETE FROM TABLENAME WHERE ID = :ROW_ID;
IF (CNT <= 0) THEN LEAVE;
END
SELECT COUNT(1) FROM TABLENAME INTO :ROW_id; /* force GC now */
END
...and every 15 minutes you do EXECUTE PROCEDURE DeleteBatch(1000).
Overall this probably would only be slower, because of single-row "precision targeting" - but at least it would spread the delays.
Use DELETE...ROWS.
https://firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-dml-delete-orderby
But as I already said in the answer to the previous question it is better to spend time investigating source of slowdown instead of workaround it by deleting data.
I am copying data from Table1 to Table2 table using sql bulk copy. I have applied trigger on Table2, but my trigger is not firing on every row. Here is my trigger and sqlbulkcopy function.
SqlConnection dstConn = new SqlConnection(ConfigurationManager.ConnectionStrings["Destination"].ConnectionString);
string destination = dstConn.ConnectionString;
//Get data from Source in our case T1
DataTable dataTable = new Utility().GetTableData("Select * From [db_sfp_ems].[dbo].[tbl_current_data_new] where [start_date]>'" + calculate_daily_Time + "' and status=0" , source);
SqlBulkCopy bulkCopy = new SqlBulkCopy(source, SqlBulkCopyOptions.FireTriggers)
{
//Add table name of source
DestinationTableName = "tbl_current_data",
BatchSize = 100000,
BulkCopyTimeout = 360
};
bulkCopy.WriteToServer(dataTable);
//MessageBox.Show("Data Transfer Succesfull.");
dstConn.Close();
------Trigger-----
ALTER TRIGGER [dbo].[trgAfterInsert] ON [dbo].[tbl_current_data]
AFTER INSERT
AS
BEGIN
declare #intime datetime
declare #sdp_id numeric
declare #value numeric(9,2)
SELECT #intime= DATEADD(SECOND, -DATEPART(SECOND, start_date), start_date) FROM INSERTED
SELECT #sdp_id= sdp_id FROM INSERTED
SELECT #value= value FROM INSERTED
INSERT INTO Table3(sdp_id,value,start_date)
VALUES
(
#sdp_id,#value,#intime
)
A trigger is fired after an insert, whether that insert concerns 0, 1 or multiple records makes no difference to the trigger. So, even though you are inserting a whole bunch of records, the trigger is only fired once. This is by design, and not specific for BULK_INSERT; this is true for every kind of insert. This also means that the inserted pseudo table can hold 0, 1 or multiple records. This is a common pitfall. Be sure to write your trigger in such a way it can handle multiple records. For example: SELECT #sdp_id= sdp_id FROM INSERTED won't work as expected if inserted holds multiple records. The variable will be set, but you cannot know what value (from which inserted record) it's going to hold.
This is all part of the set based philosophy of SQL, it is best not to try and break that philosophy by using loops or other RBAR methods. Stay in the set mindset.
Your trigger is simply broken. In SQL Server, triggers handle multiple rows at one time. Assuming that inserted has one row is fatal error -- and I wish it caused a syntax error.
I think this is the code you want:
ALTER TRIGGER [dbo].[trgAfterInsert] ON [dbo].[tbl_current_data]
AFTER INSERT
AS
BEGIN
INSERT INTO Table3 (sdp_id, value, start_date)
SELECT sdp_id, value,
DATEADD(SECOND, -DATEPART(SECOND, start_date), start_date)
FROM inserted i;
END;
Apart from being correct, another advantage is that the code is simpler to write.
Note: You are setting the "seconds" part to 0. However -- depending on the type -- start_date could have fractional seconds that remain. If that is an issue, ask another question.
It is old question but to be sure i am asking again.
Actually i have created sequence like in Oracle with a table and want to use with multiple threads and multiple JVM's all process will be hitting it parallel.
Following is sequence stored procedure just want to ask whether this will work with multiple JVM's and always provide unique number to threads in all jvm's or is there any slight chance of it returning same sequence number two more than one calls?
create table sequenceTable (id int)
insert into sequenceTable values (0)
create procedure mySequence
AS
BEGIN
declare #seqNum int
declare #rowCount int
select #rowCount = 0
while(#rowCount = 0)
begin
select #seqNum = id from sequenceTable
update sequenceTable set id = id + 1 where id = #seqNum
select #rowCount = ##rowcount
print 'numbers of rows update %1!', #rowCount
end
SELECT #seqNum
END
If you choose to maintain your current design of updating the sequenceTable.id column each time you want to generate a new sequence number, you need to make sure:
the 'current' process gets an exclusive lock on the row containing the desired sequence number
the 'current' process then updates the desired row and retrieves the newly updated value
the 'current' process releases the exclusive lock
While the above can be implemented via a begin tran + update + select + commit tran, it's actually a bit easier with a single update statement, eg:
create procedure mySequence
AS
begin
declare #seqNum int
update sequenceTable
set #seqNum = id + 1,
id = id + 1
select #seqNum
end
The update statement is its own transaction so the update of the id column and the assignment of #seqNum = id + 1 is performed under an exclusive lock within the update's transaction.
Keep in mind that the exclusive lock will block other processes from obtaining a new id value; net result is that the generation of new id values will be single-threaded/sequential
While this is 'good' from the perspective of ensuring all processes obtain a unique value, it does mean this particular update statement becomes a bottleneck if you have multiple processes hitting the update concurrently.
In such a situation (high volume of concurrent updates) you could alleviate some contention by calling the stored proc less often; this could be accomplished by having the calling processes request a range of new id values (eg, pass #increment as input parameter to the proc, then instead of id + 1 you use id + #increment), with the calling process then knowing it can use sequence numbers (#seqNum-#increment+1) to #seqNum.
Obviously (?) any process that uses a stored proc to generate 'next id' values only works if *ALL* processes a) always call the proc for a new id value and b) *ALL* processes only use the id value returned by the proc (eg, they don't generate their own id values).
If there's a possibility of applications not following this process (call proc to get new id value), you may want to consider pushing the creation of the unique id values out to the table where these id values are being inserted; in other words, modify the target table's id column to include the identity attribute; this eliminates the need for applications to call the stored proc (to generate a new id) and it (still) ensures a unique id is generated for each insert.
You can emulate sequences in ASE. Use the reserve_identity function to achieve required type of activity:
create table sequenceTable (id bigint identity)
go
create procedure mySequence AS
begin
select reserve_identity('sequenceTable', 1)
end
go
This solution is non-blocking and does generate minimal transaction log activity.
I am working with an application that generates invoice numbers (sequentially based on few parameters) and so far it has been using a trigger with serialized transaction. Because the trigger is rather "heavy" it manages to timeout execution of the insert query.
I'm now working on a solution to that problem and so far I came to the point where I have a stored procedure that do the insert and after the insert I have a transaction with isolation level serializable (which by the way applies to that transaction only or should i set it back after the transaction has been commited?) that:
gets the number
if not found do the insert into that table and if found updates the number (increment)
commits the transaction
I'm wondering whether there's a better way to ensure the number is used once and gets incrementer with the table locked (only the number tables gets locked, right?).
I read about sp_getapplock, would that be somewhat a better way to achieve my goal?
I would optimize the routine for update (and handle "insert if not there" separately), at which point it would be:
declare #number int;
update tbl
set #number = number, number += 1
where year = #year and month = #month and office = #office and type = #type;
You don't need any specific locking hints or isolation levels, SQL Server will ensure no two transactions read the same value before incrementing.
If you'd like to avoid handling the insert separately, you can:
merge into tbl
using (values (#year, #month, #office, #type)) as v(y,m,o,t)
on tbl.year = v.year and tbl.month = v.month and tbl.office = v.office and tbl.type = v.type
when not matched by target then
insert (year, month, office, type, number) values(#year, #month, #office, #type, 1)
when matched then
update set #number = tbl.number, tbl.number += 1
;
Logically this should provide the same guard against race condition as update, but for some reason I don't remember where is the proof.
If you first insert and then update you have a time window where an invalid number is set and can be observed. Further, if the 2nd transaction fails which can always happen you have inconsistent data.
Try this:
Take a fresh number in tran 1.
Insert in tran 2 with the number that was taken already
That way you might burn a number but there will never be inconsistent data.