Reading on https://cloud.google.com/bigquery/quotas I see that the cumulative time limit for a multi-statement query is 24 hours.
I have a stored procedure that exceed 24 hours because it calls other stored procedures to process a large set of data for a wide time period. The time period is every week for a 5-year span. Each week it processes several terabytes of data.
After 24 hours I get error Script exceeded timeout limit.
When the procedure fails because of the timeout error, I have a log table that tells identifies the last week_id that was ran and I restart the process from that week.
To get around the timeout error, I can call the stored procedure with a code like this:
DECLARE begin_week_id INT64;
DECLARE end_week_id INT64;
SET begin_week_id = 1;
SET end_week_id = 110;
BEGIN
CALL dev.process_data(begin_week_id, end_week_id);
EXCEPTION WHEN ERROR
THEN SET begin_week_id = (SELECT max(week_id) from dev.process_week_log);
CALL dev.process_data(begin_week_id, end_week_id);
END IF;
CALL dev.process_data(begin_week_id, end_week_id);
EXCEPTION WHEN ERROR
THEN SET begin_week_id = (SELECT max(week_id) from dev.process_week_log);
CALL dev.process_data(begin_week_id, end_week_id);
END IF;
CALL dev.process_data(begin_week_id, end_week_id);
EXCEPTION WHEN ERROR
THEN SET begin_week_id = (SELECT max(week_id) from dev.process_week_log);
CALL dev.process_data(begin_week_id, end_week_id);
END IF;
CALL dev.process_data(begin_week_id, end_week_id);
EXCEPTION WHEN ERROR
THEN SET begin_week_id = (SELECT max(week_id) from dev.process_week_log);
CALL dev.process_data(begin_week_id, end_week_id);
END IF;
END;
The problem with this is, if I put the EXCEPTION WHEN ERROR in the stored procedure however, I would still get the timeout error.
It's only when I run the EXCEPTION WHEN ERROR with the call/invoke/exec that it works.
This is not ideal because at the end of the day, I would like this to be fully automated and not require to be executed with EXCEPTION WHEN ERROR outside of the stored procedure. I will have to pass this process on to a coworker in the future and I'd prefer that it would be more seamless.
Is there a better way to get around the timeout error in BigQuery? Any help is appreciated. Thank you.
Related
I have this anonymous PL/SQL block which calculates and prints a value return from a table.
DECLARE
U_ID NUMBER :=39;
RETAIL BINARY_FLOAT:=1;
FLAG NUMBER;
BEGIN
SELECT NVL(RETAIL_AMOUNT,1),UNIT_ID INTO RETAIL, FLAG FROM UNITS WHERE UNIT_ID=U_ID;
LOOP
SELECT NVL(MAX(UNIT_ID),U_ID) INTO FLAG FROM UNITS WHERE FATHER_ID=FLAG;
IF FLAG=U_ID THEN EXIT; END IF;
SELECT RETAIL* RETAIL_AMOUNT INTO RETAIL FROM UNITS WHERE UNIT_ID=FLAG;
EXIT WHEN FLAG=U_ID;
END LOOP;
DBMS_OUTPUT.PUT_LINE( RETAIL);
END;
This block work correctly, but I wanted to do the same thing using a PL/SQL Function
I wrote the function as follow:
CREATE OR REPLACE FUNCTION GET_UNIT_RETAIL(U_ID NUMBER)
RETURN NUMBER
IS
RETAIL BINARY_FLOAT:=1;
FLAG NUMBER;
BEGIN
SELECT NVL(RETAIL_AMOUNT,1),UNIT_ID
INTO RETAIL, FLAG
FROM UNITS
WHERE UNIT_ID=U_ID;
LOOP
SELECT NVL(MAX(UNIT_ID),U_ID)
INTO FLAG
FROM UNITS
WHERE FATHER_ID=FLAG;
IF FLAG=U_ID THEN
EXIT;
END IF;
SELECT RETAIL* RETAIL_AMOUNT
INTO RETAIL
FROM UNITS
WHERE UNIT_ID=FLAG;
EXIT WHEN FLAG=U_ID;
END LOOP;
RETURN NUMBER;
END;
/
When I try to execute the above code to save the function to the database, the environment (SQL*PLUS) hangs for a long time and at the end returns this error:
ERROR at line 1:
ORA-04021: timeout occurred while waiting to lock object
What is the problem ??? Please !
Sounds like ddl_lock problem
Take a look at
dba_ddl_locks to see who is "blocking" a create or replace.
Also try to create under different name - and see what happens.
The problem was because the Object GET_UNIT_RETAIL was busy by other environment
Here is the answer:
https://community.oracle.com/thread/2321256
I have a small loop procedure that is waiting for another process to write a flag to a table. Is there any way to add a delay so this process doesn't consume so much cpu? I believe it may need to run between 1-2 min if everything ends correctly.
BEGIN
DECLARE STOPPED_TOMCAT VARCHAR (1);
UPDATE MRC_MAIN.DAYEND SET DENDSTR = 'Y';
SET STOPPED_TOMCAT = (SELECT TOMCSTP FROM MRC_MAIN.DAYEND);
WHILE ( STOPPED_TOMCAT <> 'Y')
DO
SET STOPPED_TOMCAT = (SELECT TOMCSTP FROM MRC_MAIN.DAYEND);
END WHILE;
END;
Use call dbms_alert.sleep(x), where x - number of seconds.
I don't have the resources to test this solution, but why don't try calling IBM i Command DLYJOB in your code:
CALL QCMDEXC('DLYJOB DLY(1)', 13);
The parameter DLY indicates the wait time in seconds and the number 13 is the length of the command string being executed.
Let's say I have a for loop
for i in array.first .. array.last loop
boolean := c(i) > d(i);
if boolean --is true then
exit the loop immediately and also exit the entire procedure
else if the boolean is never true til the end of the loop, exit the loop
and keep running other scripts in this procedure.
I know the 'EXIT' keyword needs to be inside of the loop in order to exit the loop if condition is met. And 'RETURN' needs to be outside of the loop, to exit the procedure.
But if I put 'RETURN' outside of the loop, then I think no matter what the result is from the loop, it will exit the entire procedure when the loop ends?
If you want to be didactic about it, you should use an EXIT to get out of the loop, then use a RETURN to exit from the procedure (duplicating any necessary tests), thus following the structured programming rule that "A procedure should only have a single entrance and a single exit". In practice 99.999% of programmers would just code the RETURN inside the body of the loop because A) it's clearer as to what's going on (you're not just getting out of the loop, you're returning from the procedure), and B) it's shorter. Do as you will. Best of luck.
define an exception, and when ever you want to exit raise and handle the exception as
create procedure exit_loop_example as
exit_loop_exception exception;
begin
/* previous code block */
begin
for i in 1..20 loop
raise exit_loop_exception;
end loop;
exception when
exit_loop_exception then
/* handle exit loop*/
null;
end;
/* next code block */
end;
The simple loop. It’s called simple for a reason: it starts simply with the LOOP keyword and ends with the END LOOP statement. The loop will terminate if you execute an EXIT, EXIT WHEN, or RETURN within the body of the loop (or if an exception is raised).
See Oracle Magazine
Following through with Tamás Kecskeméti's link, the only recommended way is to use a while loop with the desired condition specified in the beginning itself.
Below is and excerpt from the above link :
Code Listing 5: A WHILE loop with one exit
PROCEDURE display_multiple_years (
start_year_in IN PLS_INTEGER
, end_year_in IN PLS_INTEGER)
IS
l_current_year PLS_INTEGER := start_year_in;
BEGIN
WHILE ( l_current_year <= end_year_in
AND total_sales_for_year (l_current_year) > 0)
LOOP
display_total_sales_for_year (l_current_year);
l_current_year := l_current_year + 1;
END LOOP;
END display_multiple_years;
I have multiple stored procedures calling multiple stored procedures in my database. To give a small example, I've constructed a fictionalised version of a few of them below. In my example, a Java program calls calculate_bill, which calls calculate_commission, which calls update_record.
I'm hoping to get some advice on how best to propagate the error messages up the stack to the calling Java layer, so the user gets a precise error message corresponding to wherever the error has occurred.
I'm really quite stuck on this. I've played around in my example with raise_application_error and just continually shuffling it up the stack. Is the way I'm doing it below remotely correct? Or is one raise_application_error in the relevant procedure enough, with no pragma exception init etc needed?
To give an idea of what I mean, in the example below, if a user entered a number which corresponded to a record which couldn't be updated because it didn't exist, I'd like them to get the message:
"Error calculating bill. Error calculating commission. No record exists to be updated" or something to that effect.
So two questions:
What is the best practice, most efficient, most tidy way to pass error messages up the stack for the end user in the application layer?
Does anyone have any suggestions on tidier output from the code, i.e. the best way to concatenate these errors to make them more meaningful? I'm really open to any suggestions on how to make this work best as I have absolutely no prior experience in this.
Example:
(Error in code):
-20000 : Error in top level procedure
-20001 : Error in middle level procedure
-20002 : Error in bottom level procedure
Java code:
try {
// call calculate_bill
exception (SQLException ex)
// output oracle code and relevant message.
Oracle code:
create or replace procedure calculate_bill(in_num NUMBER)
is
error_calculating_commission EXCEPTION;
error_updating_record EXCEPTION;
PRAGMA EXCEPTION_INIT (error_calculating_commission, -20001);
PRAGMA EXCEPTION_INIT (error_updating_record , -20002);
begin
if in_num > 2 then
calculate_commission(in_num);
else
raise_application_error(-20000, 'Error calculating bill. ' || 'Record number doesn''t exist.', false);
end if;
exception
when error_calculating_commission then
raise_application_error(SQLCODE, 'Error calculating bill. ' || SQLERRM, false);
when error_updating_record then
raise_application_error(SQLCODE, 'Error calculating bill. ' || SQLERRM, false);
when others then
raise_application_error(-20000, 'Unknown error encountered calculating bill.', false);
end;
create or replace procedure calculate_commission(in_num NUMBER)
is
begin
if in_num < 30 then
raise_application_error(-20001, 'Number too small to calculate commission.', false);
elsif in_num >= 30 and < 40 then
declare
error_storing_record EXCEPTION;
PRAGMA EXCEPTION_INIT (error_storing_record , -20002);
begin
update_record(in_num);
exception
when error_storing_record then
raise_application_error(SQLCODE, 'Error calculating commission. ' || SQLERRM, false);
when others then
raise_application_error(-20001, 'Unknown error encountered calculating commission.', false);
else
raise_application_error(-20001, 'Number too large to calculate commission', false);
end if;
end;
create or replace procedure update_record(in_num NUMBER)
is
begin
//some SQL query with a where clause, where in_num equals something
exception
when no_data_found then
raise_application_error(-20002, 'No record exists to be updated', false);
when others then
raise_application_error(-20002, 'Unknown error encountered updating record.', false);
end if;
end;
Note: I know this example is a little contrived. I was just trying to keep it brief.
The way I would implement this is to use RAISE_APPLICATION_ERROR where the error actually originates, then leave it unhandled in the other layers (or, if you want to do logging in the database, catch it in the OTHERS section, then re-raise using RAISE rather than RAISE_APPLICATION_ERROR.
You're OTHERS sections are still perpetuating the problem raised in your previous question: when an unknown error occurs you're replacing it with a generic, unhelpful message. To paraphrase Tom Kyte, "a WHEN OTHERS that does not end in RAISE is a bug`.
With no logging in the database, I would re-write the provided code like this:
create or replace procedure calculate_bill(in_num NUMBER)
is
begin
if in_num > 2 then
calculate_commission(in_num);
else
raise_application_error(-20000, 'Error calculating bill. ' || 'Record number doesn''t exist.', false);
end if;
end;
create or replace procedure calculate_commission(in_num NUMBER)
is
begin
if in_num < 30 then
raise_application_error(-20001, 'Number too small to calculate commission.', false);
elsif in_num >= 30 and in_num < 40 then
update_record(in_num);
else
raise_application_error(-20001, 'Number too large to calculate commission', false);
end if;
end;
create or replace procedure update_record(in_num NUMBER)
is
v_stub number;
begin
select 1 into v_stub from dual where 1 = 0;
exception
when no_data_found then
raise_application_error(-20002, 'No record exists to be updated', false);
end;
Below is sample input and the stack traces they created:
exec calculate_bill(0)
ORA-20000: Error calculating bill. Record number doesn't exist.
ORA-06512: at "SCHEMANAME.CALCULATE_BILL", line 7
ORA-06512: at line 1
exec calculate_bill(10)
ORA-20001: Number too small to calculate commission.
ORA-06512: at "SCHEMANAME.CALCULATE_COMMISSION", line 5
ORA-06512: at "SCHEMANAME.CALCULATE_BILL", line 5
ORA-06512: at line 1
exec calculate_bill(35)
ORA-20002: No record exists to be updated
ORA-06512: at "SCHEMANAME.UPDATE_RECORD", line 8
ORA-06512: at "SCHEMANAME.CALCULATE_COMMISSION", line 7
ORA-06512: at "SCHEMANAME.CALCULATE_BILL", line 5
ORA-06512: at line 1
exec calculate_bill(100)
ORA-20001: Number too large to calculate commission
ORA-06512: at "SCHEMANAME.CALCULATE_COMMISSION", line 9
ORA-06512: at "SCHEMANAME.CALCULATE_BILL", line 5
ORA-06512: at line 1
If you were to add error logging to these procedures, it would be a simple matter of adding an OTHERS clause to the EXCEPTION sections:
WHEN OTHERS THEN
my_logging (SQLCODE, SQLERRM, DBMS_UTILITY.format_error_backtrace ());
RAISE;
The new exception created by your RAISE_APPLICATION_ERROR calls will be handled transparently by the RAISE.
I know you can set user profiles or set a general timeout for query.
But I wish to set timeout to a specific query inside a procedure and catch the exception, something like :
begin
update tbl set col = v_val; --Unlimited time
delete from tbl where id = 20; --Unlimited time
begin
delete from tbl; -- I want this to have a limited time to perform
exception when (timeout???) then
--code;
end;
end;
Is this possible? is there any timeout exceptions at all I can catch? per block or query? didn't find much info on the topic.
No, you can not set a timeout in pl/sql. You could use a host language for this in which you embed your sql and pl/sql.
You could do:
select * from tbl for update wait 10; --This example will wait 10 seconds. Replace 10 with number of seconds to wait
Then, the select will attempt to lock the specified rows, but if it's unsuccessful after n seconds, it will throw an "ORA-30006: resource busy; acquire with WAIT timeout expired". If lock is achieved, then you can execute your delete.
Hope that helps.
Just an idea : you could do the Delete in a DBMS_JOB.
Then create a procedure to monitor the job, which then calls :
DBMS_JOB.BROKEN(JOBID,TRUE);
DBMS_JOB.remove(JOBID);
v_timer1 := dbms_utility.get_time();
WHILE TRUE
LOOP
v_timer2 := dbms_utility.get_time();
EXIT WHEN (ABS(v_timer1 - v_timer2)/100) > 60; -- cancel after 60 sec.
END LOOP;
MariaDB> select sleep(4);
+----------+
| sleep(4) |
+----------+
| 0 |
+----------+
1 row in set (4.002 sec)
MariaDB>
See: https://mariadb.com/kb/en/sleep/