I have a table with about 1 billion rows. I'm the sole user so there's no contention on locks, etc.
I noticed that when I run something like this:
DECLARE
CURSOR cur IS SELECT col FROM table where rownum < N;
BEGIN
OPEN cur;
LOOP
dbms_output.put_line("blah")
END LOOP;
CLOSE cur;
END;
there is a lag between the time when I hit enter and the time the output begins to flow in. If N is small then it's insignificant. For large N (or no WHERE clause) this lag is on the order of hours.
I'm new to oracle as you can tell, and I assumed that cursors just keep a pointer in the table which they update on every iteration of the loop. So I didn't expect a lag proportional to the size of the table over which iteration is performed. Is this wrong? Do cursors load the entire query result prior to iterating over it?
Is there a way to iterate over a table row by row without an initial overhead?
What you are seeing is that the output from DBMS_OUTPUT.PUT_LINE is not displayed until the program has finished. It doesn't tell you anything about how fast the query returned a first row. (I assume you intended to actually fetch data in your example).
There are many ways you can monitor a session, one is like this:
DECLARE
CURSOR cur IS SELECT col FROM table;
l_col table.col%ROWTYPE;
BEGIN
OPEN cur;
LOOP
FETCH cur INTO l_col;
EXIT WHEN cur%NOTFOUND;
dbms_application_info.set_module('TEST',l_col);
END LOOP;
CLOSE cur;
END;
While that is running, from another session run:
select action from v$session where module='TEST';
You will see that the value of ACTION keeps changing as the cursor fetches rows.
I also like to monitor v$session_longops for operations deemed by the Oracle optimizer to be "long operations":
select message, time_remaining
from v$session_longops
where time_remaining > 0;
Related
I made this procedure to bulk delete data (35m records). Can you see why this pl/sql procedure runs without exiting and rows are not getting deleted ?
create or replace procedure clear_logs
as
CURSOR c_logstodel IS SELECT * FROM test where id=23;
TYPE typ_log is table of test%ROWTYPE;
v_log_del typ_log;
BEGIN
OPEN c_logstodel;
LOOP
FETCH c_logstodel BULK COLLECT INTO v_log_del LIMIT 5000;
EXIT WHEN c_logstodel%NOTFOUND;
FORALL i IN v_log_del.FIRST..v_log_del.LAST
DELETE FROM test WHERE id =v_log_del(i).id;
COMMIT;
END LOOP;
CLOSE c_logstodel;
END clear_logs;
Adding in rowid instead of column name, exit when v_delete_data.count = 0; instead of EXIT WHEN c_logstodel%NOTFOUND; and changing chunk limit to 50,000 allowed the script clear 35 million rows in 15 mins
create or replace procedure clear_logs
as
CURSOR c_logstodel IS SELECT rowid FROM test where id=23;
TYPE typ_log is table of rowid index by binary_integer;
v_log_del typ_log;
BEGIN
OPEN c_logstodel;
LOOP
FETCH c_logstodel BULK COLLECT INTO v_log_del LIMIT 50000;
exit when v_log_del.count = 0;
FORALL i IN v_log_del.FIRST..v_log_del.LAST
DELETE FROM test WHERE rowid =v_log_del(i);
exit when v_log_del.count = 0;
COMMIT;
END LOOP;
COMMIT;
CLOSE c_logstodel;
END clear_logs;
First off when using BULK COLLECT LIMIT X the %NOTFOUND takes on a slightly unexpected meaning. In this case %NOTFOUND actually means Oracle could not retrieve X rows. (I guess technically it always does you fetch the next 1 and it says it could not fill the 1 row buffer.) Just move the EXIT WHEN %NOTFOUND to after the FORALL. But there is actually no reason to retrieve the data and then delete the retrieved rows. While one statement would be considerable faster 35M rows would require signifient rollback space. There is an interment solution.
Although not commonly used Delate statements generate rownum as do selects. This value can be user to limit the number or rows processed. So to break into a given commit size just limit rownum on the delete:
create or replace procedure clear_logs
as
k_max_rows_per_interation constant integer := 50000;
begin
loop
delete
from test
where id=23
and rownum <= k_max_rows_per_interation;
exit when sql%rowcount < k_max_rows_per_interation;
commit;
end loop;
commit;
end;
As #Stilgar points out deletes are expensive, meaning slow, so their solution may be better. But this has the advantage that it does not essentially take the table completely out-of-service during the operation. NOTE: I tend to use a much larger commit interval size, generally around 400,000 - 300,000 rows. I suggest you talk with your DBA see what they think this limit should be. Remember it is their job to properly size rollback space for typical operations. If this is normal in your operation they need to set it correctly. If you can get rollback space for 35M deletes then that is the fastest you are going to get.
I am trying to create a PL/SQL procedure which needs to do a for loop through a cursor. I have read in the oracle forums that PL/SQL 8i doesn't support subscalar queries.
This is what I have so far :
DECLARE
CURSOR C1
IS
SELECT texto,id_evento,clave_evento FROM gegf.eventos_omega_rima WHERE id_evento IN
(select max(eo.id_evento) from gegf.eventos_omega_rima eo, correctivo_rima.equipos b
where eo.fecha_ins_tab > sysdate - 25/24 and eo.fecha_ins_tab < sysdate - 1/24 and upper(eo.ORIGEN) = upper(b.nodo) and upper(b.red) = 'RIMA' group by eo.clave_evento);
r_emp C1%ROWTYPE;
BEGIN
OPEN C1;
LOOP
FETCH c1 INTO r_emp;
EXIT WHEN C1%NOTFOUND;
INSERT INTO CORRECTIVO_RIMA.T_CLOB VALUES (r_emp.TEXTO);
END LOOP;
CLOSE c1;
END;
/
How could I workaround the fact that I can't use subscalar queries in the PL/SQL version I am using?
The PLS-00103 is telling you where the problem is; line 6 column 49. In this part of your query:
where eo.fecha_ins_tab > sysdate - and
... something is missing after the minus sign; presumably you're trying to subtract some number of days from today, but you haven't supplied that number.
I don't have an 8i database lying around any more (perhaps not surprisingly) but I don't recall ever needing to quote a cursor query; and if you do I'm pretty sure the semicolon would need to be outside the closing quote. But that was also what was causing the earlier line 4, column 5 error, which was pointing at that opening quote.
You will also try to insert the last value twice; you need to test C1%NOTFOUND before the INSERT, immediately after the FETCH (unless you are using bulk collect). Of course you're inserting a dummy value, but you'll get one too many rows; with your real CLOB you'd process the last fetch value twice.
Hi friends I have a perl script which interacts with oracle Db through sqlplus. Here I am running two queries. first query fetches the serial numbers for a specific date from view and writes to a file.
SELECT DISTINCT serialnumber FROM VOUCHERHISTORYCLIENTV WHERE at LIKE '&1'; (where &1 = $date)
Then perl loop reads this file, takes serial number line by line & runs the below query, writes the data to another file which then processed by the following perl code. Now these serial numbers are too huge so every iteration of loop connects to sqlplus runs the query & generates the output & then disconnects. That is why it is taking too much time.
SELECT * FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV WHERE serialnumber = '&1'; (where &1 = $serialnumber)
Is there any way where I don't need to connect & disconnect sqlplus session again & again? what I want in the end is result of the second query for each iteration appended on an unix file..so that my perl script could format them. I guess pl/sql loop can do that...but I have never worked on pl/sql...could you please help me here?
PS: I can't use DBD::oracle here as I am facing too many issues while installing this module on solaris mahine & as this server is 3rd party server so I can't make any chnages on this
Update: 1
I tried below two procedures & run for specific date for which table has 6 million records. but query kept running & didn't produce any output even after 2 days...
Procedure# 1:
DECLARE
CURSOR CUR1 IS
SELECT DISTINCT serialnumber FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV WHERE at LIKE '&1';
CURSOR CUR2(p_ser_num NUMBER) IS
SELECT serialnumber, state, at as time1, operatorid FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV WHERE serialnumber = p_ser_num;
BEGIN
FOR l_cur1 IN CUR1
LOOP
NULL;
FOR l_cur2 IN CUR2(l_cur1.serialnumber)
LOOP
DBMS_OUTPUT.PUT_LINE(l_cur2.serialnumber||' '||l_cur2.state ||' '||l_cur2.time1 ||' '||l_cur2.operatorid );
END LOOP;
END LOOP;
END;
/
quit;
Procedure# 2:
DECLARE
CURSOR CUR1 IS
SELECT DISTINCT serialnumber FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV WHERE at LIKE '&1';
TYPE t_tab1 IS TABLE OF CUR1%ROWTYPE;
l_tab1 t_tab1;
CURSOR CUR2(p_ser_num NUMBER) IS
SELECT serialnumber, state, at as time1, operatorid FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV WHERE serialnumber = p_ser_num;
TYPE t_tab2 IS TABLE OF CUR2%ROWTYPE;
l_tab2 t_tab2;
BEGIN
OPEN CUR1;
LOOP
FETCH CUR1 BULK COLLECT INTO l_tab1;
EXIT WHEN CUR1%NOTFOUND;
END LOOP;
CLOSE CUR1;
FOR i in 1..l_tab1.COUNT
LOOP
OPEN CUR2(l_tab1(i).serialnumber);
LOOP
FETCH CUR2 BULK COLLECT INTO l_tab2;
EXIT WHEN CUR2%NOTFOUND;
END LOOP;
CLOSE CUR2;
for j in 1..l_tab2.COUNT
LOOP
DBMS_OUTPUT.PUT_LINE(l_tab2(j).serialnumber||' '||l_tab2(j).state ||' '||l_tab2(j).time1 ||' '||l_tab2(j).operatorid );
END LOOP;
END LOOP;
END;
/
quit;
Can I improve the above procedures or there is any other way which can do the job for us? Please help.
For once, you can just replace the 2 queries with this single query:
SELECT * FROM VSOWNERUPE07.VOUCHERHISTORYCLIENTV
WHERE serialnumber IN (SELECT DISTINCT serialnumber FROM
VOUCHERHISTORYCLIENTV WHERE at LIKE '&1');
And write to output to your final file.
You see any issues with this?
UPDATE1
First, breaking it into 2 different queries is almost never a better solution.SQL context switch will kill you.
Build index on serialnumber and at columns of the two tables respectively and try with the single query.
If even then its slow you can try UPDATE2.
UPDATE2
If too many records is the problem,, then use Bulk Processing.
The process is a bit tedious, but is designed to handle these kind of situations.
Some examples how to do Bulk Processing are here, here and here.
For this to work, you will probably have to write a PLSQL procedure.
Inside the procedure, bulk collect all the data into a , say, varray from your first query.
now run the second select query using this varray and collect your data.
Next
You can put these serial numbers into, say another v array and write it to a file
or
create a new single column table and write it to this new table.
then write it to a file/query this new table from perl separately.
Note1:
If you are using a separate table, truncate it after each run; and build index on it
Note2: If you are clueless of what I am talking about, check out those links first.
Suppose I have something like:
CURSOR foo_cur IS
SELECT * FROM foo;
...
DELETE FROM foo WHERE bar=1;
FOR row IN foo_cur
LOOP
...
END LOOP;
If I delete rows from foo before I open the cursor, will these rows still be part of the cursor result? Is the query SELECT * FROM foo executed at the line FOR row IN foo_cur?
The set of rows that will be returned from a cursor is determined at the point that the cursor is opened (either via an explicit OPEN or implicitly by the FOR loop). In this case, the row(s) that you deleted will not be returned in your loop.
Generally, the query is not executed all at once. Oracle executes the query enough to fetch the next set of rows, returns those rows to the PL/SQL VM, and waits until the request comes to fetch more rows. In 11g, Oracle will do an implicit BULK COLLECT of 100 rows at a time so every 100 iterations of the loop the query is executed further until there are no more rows to return. Because of multi-version read consistency, Oracle will always return the data to you as it existed when the cursor was opened even if other sessions are making and committing changes while your code is running.
I am fairly new to SQL and was wondering if someone can help me.
I got a database that has around 10 million rows.
I need to make a script that finds the records that have some NULL fields, and then updates it to a certain value.
The problem I have from doing a simple update statement, is that it will blow the rollback space.
I was reading around that I need to use BULK COLLECT AND FETCH.
My idea was to fetch 10,000 records at a time, update, commit, and continue fetching.
I tried looking for examples on Google but I have not found anything yet.
Any help?
Thanks!!
This is what I have so far:
DECLARE
CURSOR rec_cur IS
SELECT DATE_ORIGIN
FROM MAIN_TBL WHERE DATE_ORIGIN IS NULL;
TYPE date_tab_t IS TABLE OF DATE;
date_tab date_tab_t;
BEGIN
OPEN rec_cur;
LOOP
FETCH rec_cur BULK COLLECT INTO date_tab LIMIT 1000;
EXIT WHEN date_tab.COUNT() = 0;
FORALL i IN 1 .. date_tab.COUNT
UPDATE MAIN_TBL SET DATE_ORIGIN = '23-JAN-2012'
WHERE DATE_ORIGIN IS NULL;
END LOOP;
CLOSE rec_cur;
END;
I think I see what you're trying to do. There are a number of points I want to make about the differences between the code below and yours.
Your forall loop will not use an index. This is easy to get round by using rowid to update your table.
By committing after each forall you reduce the amount of undo needed; but make it more difficult to rollback if something goes wrong. Though logically your query could be re-started in the middle easily and without detriment to your objective.
rowids are small, collect at least 25k at a time; if not 100k.
You cannot index a null in Oracle. There are plenty of questions on stackoverflow about this is you need more information. A functional index on something like nvl(date_origin,'x') as a loose example would increase the speed at which you select data. It also means you never actually have to use the table itself. You only select from the index.
Your date data-type seems to be a string. I've kept this but it's not wise.
If you can get someone to increase your undo tablespace size then a straight up update will be quicker.
Assuming as per your comments date_origin is a date then the index should be on something like:
nvl(date_origin,to_date('absolute_minimum_date_in_Oracle_as_a_string','yyyymmdd'))
I don't have access to a DB at the moment but to find out the amdiOaas run the following query:
select to_date('0001','yyyy') from dual;
It should raise a useful error for you.
Working example in PL/SQL Developer.
create table main_tbl as
select cast( null as date ) as date_origin
from all_objects
;
create index i_main_tbl
on main_tbl ( nvl( to_date(date_origin,'yyyy-mm-dd')
, to_date('0001-01-01' ,'yyyy-mm-dd') )
)
;
declare
cursor c_rec is
select rowid
from main_tbl
where nvl(date_origin,to_date('0001-01-01','yyyy-mm-dd'))
= to_date('0001-01-01','yyyy-mm-dd')
;
type t__rec is table of rowid index by binary_integer;
t_rec t__rec;
begin
open c_rec;
loop
fetch c_rec bulk collect into t_rec limit 50000;
exit when t_rec.count = 0;
forall i in t_rec.first .. t_rec.last
update main_tbl
set date_origin = to_date('23-JAN-2012','DD-MON-YYYY')
where rowid = t_rec(i)
;
commit ;
end loop;
close c_rec;
end;
/