I got a bit of a problem with a Oracle query
create or replace
PROCEDURE "LOAD_USERNAME"
IS
cursor usu is
select userid
from local_user;
BEGIN
for usu_rec in usu
loop
update loc_user set username =(
select cod_user
from (
select cod_user, row_number() over (order by date_createad desc) r
from user_table#DBLINK where cod_person=usu_rec.userid
)
where r = 1
)
where externaluserid=usu_rec.userid;
end loop;
END;
Basically, trying to get code of a user(last one created) from other database and update a local table. This seems to work but I takes too much time. I can only check the remote database through a DBLINK.
Please, I want some help for a better way to do this.
I thank your help in advance.
You want to minimise the number of times you go over the network. So you should join to the remote table in your driving cursor and pull the username back there. This will be better as that query is only executed once (indexing/design will determine how well it goes). But your updates will then only be working with local data.
Edit: Removed my PL/SQL as #Aitor's was better
As Sodved said, is better to had the join in your cursor. You can try something like this:
create or replace
PROCEDURE "LOAD_USERNAME"
IS
cursor usu is
select distinct local_user.userid,your_dblink_table.cod_user
from local_user, user_table#bdlink your_dblink_table
where local_user.userid=your_dblink_table.codperson
and local_user.externaluserid=local_user.userid;
BEGIN
for usu_rec in usu
loop
update loc_user set username =usu_rec.cod_user
where externauserid=usu_rec.userid;
end loop;
commit;
END;
If you have to load massive updates, you can try a bulk collect/for all approach in the cursor.
Oracle has provided built-in functionality for this sort of thing for several major versions. If you're on an older database you should use replication. In more recent versions this has been deprecated in favour of Streams.
Related
I need to delete a bunch of records (literally millions) but I don't want to make it in an individual statement, because of performance issues. So I created a view:
CREATE VIEW V1
AS
SELECT FIRST 500000 *
FROM TABLE
WHERE W_ID = 14
After that I do a bunch deletes for example:
DELETE FROM V1 WHERE TS < 2021-01-01
What I want is to import this logic in a While loop and in stored procedure. I tried SELECT COUNT query like this:
SELECT COUNT(*)
FROM TABLE
WHERE W_ID = 14 AND TS < 2021-01-01;
Can I use this number in the same procedure as a condition and how can I manage that?
This is what I have tried and I get an error
ERROR: Dynamic SQL Error; SQL error code = -104; Token unknown; WHILE
Code:
CREATE PROCEDURE DeleteBatch
AS
DECLARE VARIABLE CNT INT;
BEGIN
SELECT COUNT(*) FROM TABLE WHERE W_ID = 14 AND TS < 2021-01-01 INTO :cnt;
WHILE cnt > 0 do
BEGIN
IF (cnt > 0) THEN
DELETE FROM V1 WHERE TS < 2021-01-01;
END
ELSE break;
END
I just can't wrap my head around this.
To clarify, in my previous question I wanted to know how to manage the garbage_collection after many deleted records, and I did what was suggested - SELECT * FROM TABLE; or gfix -sweep and that worked very well. As mentioned in the comments the correct statement is SELECT COUNT(*) FROM TABLE;
After that another even bigger database was given to me - above 50 million. And the problem was the DB was very slow to operate with. And I managed to get the server it was on, killed with a DELETE statement to clean the database.
That's why I wanted to try deleting in batches. The slow-down problem there was purely hardware - HDD has gone, and we replaced it. After that there was no problem with executing statements and doing backup and restore to reclaim disk space.
Provided the data that you need to delete, doesn't ever need to be rollbacked once the stored procedure is kicked off, there is another way to handle massive DELETEs in a Stored Procedure.
The example stored procedure will delete the rows 500,000 at a time. It will loop until there aren't any more rows to delete. The AUTONOMOUS TRANSACTION will allow you to put each delete statement in its own transaction and it will commit immediately after the statement completes. This is issuing an implicit commit inside a stored procedure, which you normally can't do.
CREATE OR ALTER PROCEDURE DELETE_TABLEXYZ_ROWS
AS
DECLARE VARIABLE RC INTEGER;
BEGIN
RC = 9999;
WHILE (RC > 0) DO
BEGIN
IN AUTONOMOUS TRANSACTION DO
BEGIN
DELETE FROM TABLEXYZ ROWS 500000;
RC = ROW_COUNT;
END
END
SELECT COUNT(*)
FROM TABLEXYZ
INTO :RC;
END
because of performance issues
What are those exactly? I do not think you actually are improving performance, by just running delete in loops but within the same transaction, or even different TXs but within the same timespan. You seem to be solving some wrong problem. The issue is not how you create "garbage", but how and when Firebird "collects" it.
For example, Select Count(*) in Interbase/Firebird engines means natural scan over all the table and the garbage collection is often trigggered by it, which can itself get long if lot of garbage was created (and massive delete surely does, no matter if done by one million-rows statement or million of one-row statements).
How to delete large data from Firebird SQL database
If you really want to slow down deletion - you have to spread that activity round the clock, and make your client application call a deleting SP for example once every 15 minutes. You would have to add some column to the table, flagging it is marked for deletion and then do the job like that
CREATE PROCEDURE DeleteBatch(CNT INT)
AS
DECLARE ROW_ID INTEGER;
BEGIN
FOR SELECT ID FROM TABLENAME WHERE MARKED_TO_DEL > 0 INTO :row_id
DO BEGIN
CNT = CNT - 1;
DELETE FROM TABLENAME WHERE ID = :ROW_ID;
IF (CNT <= 0) THEN LEAVE;
END
SELECT COUNT(1) FROM TABLENAME INTO :ROW_id; /* force GC now */
END
...and every 15 minutes you do EXECUTE PROCEDURE DeleteBatch(1000).
Overall this probably would only be slower, because of single-row "precision targeting" - but at least it would spread the delays.
Use DELETE...ROWS.
https://firebirdsql.org/file/documentation/html/en/refdocs/fblangref25/firebird-25-language-reference.html#fblangref25-dml-delete-orderby
But as I already said in the answer to the previous question it is better to spend time investigating source of slowdown instead of workaround it by deleting data.
I'm trying to figure out how to create a looped truncate to truncate (or delete) the data in specific tables only. There are something like 30 or so tables that need to be truncated, and I want to avoid just using a list of truncate statements. I can't seem to find any good examples of doing this, and no matter what I try, I get a "right truncated" error.
Example 1:
FOR anlyc_tables AS curs CURSOR FOR
SELECT table_name FROM systable WHERE table_name LIKE 'table_to_truncate_prefix%'
DO EXECUTE (
'TRUNCATE TABLE ' + table_name
);
END FOR;
This one throws no errors, but completes in .078 seconds and doesn't actually truncate anything.
Example 2:
ALTER PROCEDURE truncate_analytics()
BEGIN
DECLARE #table_name VARCHAR;
DECLARE curs DYNAMIC SCROLL CURSOR FOR SELECT table_name FROM systable WHERE table_name LIKE 't_anlyc%';
OPEN curs WITH HOLD;
FETCH NEXT curs INTO #table_name;
WHILE(sqlstate = 0) LOOP
FETCH NEXT curs INTO #table_name;
TRUNCATE TABLE table_name;
END LOOP;
END
GO
CALL truncate_analytics()
GO
Results in the "right truncated" error and the tables are not truncated.
I think I'm missing something really obvious here, but I don't have a ton of experience with SQL scripting in this manner, and can't seem to find any working examples of this to prove it's even possible.
Can anyone point me in the right direction?
There may be a few things at work that are preventing the code from working.
With your second example, the text "TRUNCATE TABLE table_name" is going to look for a table named table_name. table_name won't be treated for it's variable value, but will be considered a object identifier. FWIW, I'm not sure how this would cause a right truncation error.
Depending on what you consider the prefix for a table, you might not be getting the list of tables you expect. If you are considering the user to be the prefix (i.e. dba.my_table), your select isn't going to grab the necessary tables; there can be multiple tables with the same name, just different owners. If you want to select based on user, you'll need to join to the SYSUSER view.
Lastly, using a cursor for this can be done. It would require WITH HOLD as you already have, because the TRUNCATE TABLE statement issues an implicit commit and closes cursors otherwise. (Note that WITH HOLD requires explicit closing of the cursor, otherwise it will remain open until connection close.) Another option is to build a string of statements to execute, and use the execute or EXECUTE IMMEDIATE on them as a batch.
Hope This Helps,
Tyson
I'm trying to create a stored procedure (or a a trigger, function, anything) to check if all the tables in a database is accessible.
My idea is to get all the tables in database, and then try to acess them with a simple select, if this succeeds for all the tables, everything is supposed to be ok.
I couldn't think of anything else to solve this problem, but I don't know how to do this, or all this intead.
1 - to get all the tables name I did:
SELECT RDB$RELATION_NAME TABLE
FROM RDB$RELATIONS
WHERE RDB$VIEW_BLR IS NULL
AND (RDB$SYSTEM_FLAG IS NULL OR RDB$SYSTEM_FLAG = 0)
ORDER BY TABLE
Now I just need to create the SELECT statement to each table, and run a query:
SELECT FIRST 1 * FROM [TABLE];
while it's ok, it continues, if all the tables are accessible, my database is ok.
Can anybody help me with this? Is this the correct aproach to solve this problem?
As a_horse_with_no_name commented, this is really strange request... if you see the table in the RDB$RELATIONS you can be pretty sure the table exists in the database. If the table is listed in the DB metadata but actually doesn't exist then the DB is corrupted and your idea to use select to check it's "accessibilty" is pointless... Also, the table might be there but the user might not have select right for it, IOW you need to take the user rights into account too.
Anyway, you can use the EXECUTE STATEMENT to execute dynamically built DSQL statement, something like
declare stmt varchar(1024);
declare ctab varchar(31);
BEGIN
FOR SELECT RDB$RELATION_NAME
FROM RDB$RELATIONS
WHERE RDB$VIEW_BLR IS NULL AND (RDB$SYSTEM_FLAG IS NULL OR RDB$SYSTEM_FLAG = 0)
INTO :ctab DO BEGIN
stmt = 'select ... from ' || ctab;
execute statement stmt;
END;
END
To check is the database corrupted you should use the gfix utility with -validate option.
How can I generate insert and update stored procedures for specific or all tables automatically in SQL Server ?
Is it possible ? thanks in advance
There are tons of tools that can do this for you, some free, some that you pay for. Here is a simple one with source you can adapt:
http://www.codeproject.com/Articles/19280/Stored-Procedure-Generator
and here is an SSMS addin, that is also free (or for a small fee depending on the version you choose).
http://www.ssmstoolspack.com/
-- It is for std table
create proc std_delete
#id int
as begin
delete from std where id=#id
end
execute std_delete 2
-- SP for delete
--Now u can c
select * from std
I'm trying to find a way to do a conditional DELETE on an InnoDB table which contains millions of records, without locking it (thus not bringing the website down).
I've tried to find information on mysql.com, but to no avail. Any tips on how to proceed?
I don't think it is possible to delete without locking. That said, I don't think locking the record you want to delete is a problem. What would be a problem is locking other rows.
I found some information on that subject here: http://dev.mysql.com/doc/refman/5.0/en/innodb-locks-set.html
What I would suggest, is to try and do a million single row deletes. I think that if you do all those in a single transaction, performance should not hurt too much. so you would get something like:
START TRANSACTION;
DELETE FROM tab WHERE id = 1;
..
..
DELETE FROM tab WHERE id = x;
COMMIT;
You can generate the required statments by doing something like
SELECT CONCAT('DELETE FROM tab WHERE id = ', id)
FROM tab
WHERE <some intricate condition that selects the set you want to delete>
So the advantage over this method instead of doing:
DELETE FROM tab
WHERE <some intricate condition that selects the set you want to delete>
is that in the first approach you only ever lock the record you're deleting, whereas in the second approach you could run the risk of locking other records that happen to be in the same range as the rows you are deleteing.
If it fits your application, then you could limit the number of rows to delete, and setup a cronjob for repeating the deletion. E.g.:
DELETE FROM tab WHERE .. LIMIT 1000
I found this to be good compromise in a similar scenario.
I use procedure to delete
create procedure delete_last_year_data()
begin
DECLARE del_row varchar(255);
DECLARE done INT DEFAULT 0;
declare del_rows cursor for select CONCAT('DELETE FROM table_name WHERE id = ', id)
from table_name
where created_time < '2018-01-01 00:00:00';
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1;
open del_rows;
repeat
fetch del_rows into del_row;
if not done
then
set #del = del_row;
prepare stmt from #del;
execute stmt;
DEALLOCATE PREPARE stmt;
end if;
until done end repeat;
close del_rows;
end //