I have a fairly time intensive PL/SQL block that builds fingerprints from molecular structures. I would like to print output to SQL*Plus console to provide feedback on how many structures have been processed. I can do this with dbms_output.put_line
However everytime that is called a new line is written. I want to overwrite the line.
For example, currently I have the below.
Structure x of y processed
Structure x of y processed
Structure x of y processed
Structure x of y processed
Eventually I fill up the buffer as I'm dealing with thousands of structure records.
Is there a method I can use that will just overwrite the last output line?
Using DBMS_OUTPUT means that SQL*Plus will display nothing until the entire PL/SQL block is complete and will then display all the data currently in the buffer. It is not, therefore, an appropriate way to provide an ongoing status.
On the other hand, Oracle does provide a package DBMS_APPLICATION_INFO that is specifically designed to help you monitor your running code. For example, you could do something like
CREATE PROCEDURE process_structures
AS
<<other variable declarations>>
rindex BINARY_INTEGER;
slno BINARY_INTEGER;
totalwork NUMBER := y; -- Total number of structures
worksofar NUMBER := 0; -- Number of structures processed
BEGIN
rindex := dbms_application_info.set_session_longops_nohint;
FOR i IN (<<select structures to process>>)
LOOP
worksofar := worksofar + 1;
dbms_application_info.set_session_longops(
rindex => rindex,
slno => slno,
op_name => 'Processing of Molecular Structures',
sofar => worksofar ,
totalwork => totalwork,
target_desc => 'Some description',
units => 'structures');
<<process your structure with your existing code>>
END LOOP;
END;
From a separate SQL*Plus session, you can then monitory progress by querying the V$SESSION_LONGOPS view
SELECT opname,
target_desc,
sofar,
totalwork,
units,
elapsed_seconds,
time_remaining
FROM v$session_longops
WHERE opname = 'Processing of Molecular Structures';
You may also send messages to a named pipe and have another process read the message from the pipe.
procedure sendmessage(p_pipename varchar2
,p_message varchar2) is
s number(15);
begin
begin
sys.dbms_pipe.pack_message(p_message);
exception
when others then
sys.dbms_pipe.reset_buffer;
end;
s := sys.dbms_pipe.send_message(p_pipename, 0);
if s = 1
then
sys.dbms_pipe.purge(p_pipename);
end if;
end;
function receivemessage(p_pipename varchar2
,p_timeout integer) return varchar2 is
n number(15);
chr varchar2(200);
begin
n := sys.dbms_pipe.receive_message(p_pipename, p_timeout);
if n = 1
then
return null;
end if;
sys.dbms_pipe.unpack_message(chr);
return(chr);
end;
I don't think you can. As far as I understood the dbms_output it just doesn't work that way.
I recommend you use put to echo a single dot and a newline every 1000 or so entries to see that something is happening and write into a table or sequence the current position so you can have a look if you want to know.
Related
I have this anonymous PL/SQL block which calculates and prints a value return from a table.
DECLARE
U_ID NUMBER :=39;
RETAIL BINARY_FLOAT:=1;
FLAG NUMBER;
BEGIN
SELECT NVL(RETAIL_AMOUNT,1),UNIT_ID INTO RETAIL, FLAG FROM UNITS WHERE UNIT_ID=U_ID;
LOOP
SELECT NVL(MAX(UNIT_ID),U_ID) INTO FLAG FROM UNITS WHERE FATHER_ID=FLAG;
IF FLAG=U_ID THEN EXIT; END IF;
SELECT RETAIL* RETAIL_AMOUNT INTO RETAIL FROM UNITS WHERE UNIT_ID=FLAG;
EXIT WHEN FLAG=U_ID;
END LOOP;
DBMS_OUTPUT.PUT_LINE( RETAIL);
END;
This block work correctly, but I wanted to do the same thing using a PL/SQL Function
I wrote the function as follow:
CREATE OR REPLACE FUNCTION GET_UNIT_RETAIL(U_ID NUMBER)
RETURN NUMBER
IS
RETAIL BINARY_FLOAT:=1;
FLAG NUMBER;
BEGIN
SELECT NVL(RETAIL_AMOUNT,1),UNIT_ID
INTO RETAIL, FLAG
FROM UNITS
WHERE UNIT_ID=U_ID;
LOOP
SELECT NVL(MAX(UNIT_ID),U_ID)
INTO FLAG
FROM UNITS
WHERE FATHER_ID=FLAG;
IF FLAG=U_ID THEN
EXIT;
END IF;
SELECT RETAIL* RETAIL_AMOUNT
INTO RETAIL
FROM UNITS
WHERE UNIT_ID=FLAG;
EXIT WHEN FLAG=U_ID;
END LOOP;
RETURN NUMBER;
END;
/
When I try to execute the above code to save the function to the database, the environment (SQL*PLUS) hangs for a long time and at the end returns this error:
ERROR at line 1:
ORA-04021: timeout occurred while waiting to lock object
What is the problem ??? Please !
Sounds like ddl_lock problem
Take a look at
dba_ddl_locks to see who is "blocking" a create or replace.
Also try to create under different name - and see what happens.
The problem was because the Object GET_UNIT_RETAIL was busy by other environment
Here is the answer:
https://community.oracle.com/thread/2321256
We have kiosks for customers to check their purchase volume for two different categories of items. They will input their mobile number, which will send an OTP to their mobile numbers and they will input it back to authenticate, the system has to check the data and display for them. As a developer, the kiosk supplier has provided us with a limited functionality development kit by which we can execute select statement on the database and display the returned values on the kiosk.
I have created an object type as follows:
CREATE OR REPLACE TYPE rebate_values
AS
OBJECT (ASales_total number,
ACurrent_Rebate_Percent number,
ANeeded_Sales number,
ANext_Rebate_Percent number,
BSales_total number,
BCurrent_Rebate_Percent number,
BNeeded_Sales number,
BNext_Rebate_Percent number);
A function to which I will pass customers' mobile to get their sales and rebate information:
CREATE OR REPLACE FUNCTION AA_rebate_function (P_phone IN NUMBER)
RETURN rebate_values
IS
A_P_Sales_total NUMBER;
A_P_Current_Rebate_Percent NUMBER;
A_P_Needed_Sales NUMBER;
A_P_Next_Rebate_Percent NUMBER;
B_P_Sales_total NUMBER;
B_P_Current_Rebate_Percent NUMBER;
B_P_Needed_Sales NUMBER;
B_P_Next_Rebate_Percent NUMBER;
P_CODE VARCHAR (10);
BEGIN
SELECT CC_CODE
INTO P_CODE
FROM CUSTOMERS
WHERE C_MOBILE = P_phone;
FOR OUTDATA
IN (
--My Query to retrieve the data
Select ................
)
LOOP
IF OUTDATA.CLASS = 'X'
THEN
A_P_Sales_total := OUTDATA.SALES_TOTAL;
A_P_Current_Rebate_Percent := OUTDATA.CURRENT_REBATE_PERCENT;
A_P_Needed_Sales := OUTDATA.NEEDED_SALES_FOR_HIGHER_REBATE;
A_P_Next_Rebate_Percent := OUTDATA.NEXT_HIGHER_REBATE_PERCENT;
END IF;
IF OUTDATA.CLASS = 'Y'
THEN
B_P_Sales_total := OUTDATA.SALES_TOTAL;
B_P_Current_Rebate_Percent := OUTDATA.CURRENT_REBATE_PERCENT;
B_P_Needed_Sales := OUTDATA.NEEDED_SALES_FOR_HIGHER_REBATE;
B_P_Next_Rebate_Percent := OUTDATA.NEXT_HIGHER_REBATE_PERCENT;
END IF;
END LOOP;
RETURN rebate_values (A_P_Sales_total,
A_P_Current_Rebate_Percent,
A_P_Needed_Sales,
A_P_Next_Rebate_Percent,
B_P_Sales_total,
B_P_Current_Rebate_Percent,
B_P_Needed_Sales,
B_P_Next_Rebate_Percent);
END;
/
The query takes 27 seconds to retrieve the values for each customer. Each customer will have 2 rows, so that's why I have used LOOP to collect the values.
When I execute the function:
SELECT AA_rebate_function (XXXXXXXXXX) FROM DUAL;
I get data as follows in a single column within 27 seconds:
(XXXX, X, XXXX, X, XXXX, X, XXXX, X)
But when I execute the function to get the values in different columns, it takes 27 x 8 seconds = 216 seconds, i.e., approximately 3.6 minutes which is a big issue as the customer cannot wait for 3.6 minutes on the kiosk to view the data.
SELECT x.c.ASales_total,
x.c.ACurrent_Rebate_Percent,
x.c.ANeeded_Sales,
x.c.ANext_Rebate_Percent,
x.c.BSales_total,
x.c.BCurrent_Rebate_Percent,
x.c.BNeeded_Sales,
x.c.BNext_Rebate_Percent
FROM (SELECT AA_rebate_function (XXXXXXXXXX) c FROM DUAL) x;
I have tried using stored procedure with OUT values but it doesn't fit in my environment as I cannot program to execute stored procedures from the kiosk development toolkit because it only supports select statements, checked with the supplier and they don't have any plan to add that support in near future.
I tried converting the single field into multiple columns using REGEXP_SUBSTR but I get a type conversion error as it is an array.
The query is very complex and has to calculate data for the last 10 years and has millions of rows, 27 seconds is actually the optimum time to get the desired results.
Interesting! I didn't realize that when you query a function that returns an object, it runs the function once for each column you reference the object in. That's awkward.
The easiest solution I could find for this is to switch your function to be PIPELINED. You'll need to create a nested table type to do this.
create type rebate_values_t is table of rebate_values;
/
CREATE OR REPLACE FUNCTION AA_rebate_function (P_phone IN NUMBER)
RETURN rebate_values_t PIPELINED
IS
... your code here ...
PIPE ROW (rebate_values (A_P_Sales_total,
A_P_Current_Rebate_Percent,
A_P_Needed_Sales,
A_P_Next_Rebate_Percent,
B_P_Sales_total,
B_P_Current_Rebate_Percent,
B_P_Needed_Sales,
B_P_Next_Rebate_Percent));
RETURN;
END;
/
SELECT x.ASales_total,
x.ACurrent_Rebate_Percent,
x.ANeeded_Sales,
x.ANext_Rebate_Percent,
x.BSales_total,
x.BCurrent_Rebate_Percent,
x.BNeeded_Sales,
x.BNext_Rebate_Percent
FROM TABLE(AA_rebate_function (XXXXXXXXXX)) x;
For some reason, this should only execute the function once, and take 27 seconds.
I have below code in my PL/SQL procedure, which I called in API_XXX.put(it calls utl_file.put) in a while loop. And the l_xmldoc is CLOB from a function of getReportXML, which returns the xml clob.
the code I write to write xml into a file is like:
l_offset := 1;
WHILE (l_offset <= l_length)
LOOP
l_char := dbms_lob.substr(l_xmldoc,1,l_offset);
IF (l_char = to_char(10)) ---I also tried if (l_char=chr(10)) but it did not work
THEN
API_XXXX.new_line(API_XXX.output, 1);
ELSE
API_XXXX.put(fnd_API_XXX.output, l_char);
END IF;
l_offset := l_offset + 1;
END LOOP;
Please note that the API_XXX is the existing package which I am not able to modify, and this api calls fflush in the end of put.
API_XXX.put's part is like below("WHICH" is the first param):
elsif WHICH = API_XXX.OUTPUT then
temp_file := OUT_FNAME;
utl_file.put(F_OUT, BUFF);
utl_file.fflush(F_OUT);
API_XXX.new_line is like(LINES is the number of lines to write):
elsif WHICH = API_XXX.OUTPUT then
temp_file := OUT_FNAME;
utl_file.new_line(F_OUT, LINES);
utl_file.fflush(F_OUT);
I notice a that the put/new_line procedure in my customer's side will sometimes raise UTL_FILE.WRITE_ERROR for unknown reason(maybe due to the l_length is too large(up to 167465)) in the while loop from my customer.
I read Oracle PL/SQL UTL_FILE.PUT buffering
. And I found that this is the same cause, my l_xmldoc is really large and when I loop it, I found that it is without a new line terminator so the buffer is up to 32767 even though I fflush every time.
So, how should I convert the l_xmldoc into a varchar with new line terminator.
PS: I confirmed that my customer is using Oralce 11g
Post the Oracle Version you are using! Or we can just guess around...
Your fflush will not work as you expect - From the documentation:
FFLUSH physically writes pending data to the file identified by the file handle. Normally, data being written to a file is buffered. The FFLUSH procedure forces the buffered data to be written to the file. The data must be terminated with a newline character.
tbone is abolutely right the line TO_CHAR(10) is wrong! Just try SELECT TO_CHAR(10) FROM DUAL; you will get 10 which you then compare to a single character. A single character will never be '10' since 10 has two characters!
Your problem is most likely a buffer-overflow with too large XML-Files, but keep in mind, also other problems on the target system can lead to write_errors, which should be handled.
Solutions
Quick&Dirty: Since you don't seem to care about performance anyways you can just close the file every X byte and reopen it with A for append. So just add to the loop:
IF MOD( l_offset, 32000 ) = 0
THEN
UTL_FILE.FCLOSE( f_out );
UTL_FILE.FOPEN( out_fpath, out_fname, f_out, 'a', 32767 );
END IF;
Use the right tool for the right job: UTL_FILE is not suited for handling complex data. The only usecase for UTL_FILE are small newline-separated lines of text. For everything else you should write RAW bytes! (Which will also allow you porper control over ENCODING, which is currently just mini-vanilly-lucky-guess)
Write a Java-Stored-Procedure with NIO-Filechannels - fast, safe, nice... But be careful, your program might run 10 times as fast!
Just a guess, but instead of "to_char(10)" you might try chr(10) to determine/write a newline. Not sure if this will solve your problem, but sometimes very long lines (without newlines) can cause issues.
For example:
declare
l_clob clob;
l_char char;
begin
l_clob := 'Line 1' || chr(10) || 'Line 2' || chr(10);
for i in 1 .. DBMS_LOB.GETLENGTH(l_clob)
loop
l_char := dbms_lob.substr(l_clob, 1, i);
if (l_char = chr(10)) then
--if (l_char = to_char(10)) then
dbms_output.put_line('Found a newline at position ' || i);
end if;
end loop;
end;
Notice the difference between chr(10) and to_char(10). Easy enough to test if this solves your problem anyway.
I have to move around 50+ validation functions into Oracle. I'm looking for the approach that runs fastest, but also would like to get around a boolean issue if possible. The return object for them all needs to be the same so that the application can react off the result in a consistent fashion and alert the user or display whatever popups, messages we may need. I created a valObj for this, but not sure yet if that is the best approach. The return format can be changed because the front-end that reacts off of it is not developed yet. In the end it will contain many different validation functions, from integer, number, phone, email, IPv4, IPv6, etc... This is what I have so far...
/***
This is the validation object.
It stores 1 for valid, 0 for not valid and some helper text that can be relayed back to the user.
***/
create or replace type valObj as object (
result number(1),
resultText varchar(32000)
);
/***
Coming from ColdFusion this seems clean to me but the function
will end up being a couple thousand lines long.
***/
create or replace function isValid(v in varchar2, format in varchar2)
return valObj
is
test number;
begin
if format = 'number' then
begin
test := to_number(v);
return valObj(1,null);
exception when VALUE_ERROR then return valObj(0,'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...');
end;
elsif format = 'integer' then
null; --TO DO
elsif format = 'email' then
null; --TO DO
elsif format = 'IPv4' then
null; --TO DO
elsif format = 'IPv6' then
null; --TO DO
end if;
--dozens of others to follow....
end;
/
/* Example Usage in SQL */
select isValid('blah','number') from dual; -- returns: (0, Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...)
select isValid('blah','number').result from dual; -- returns: 0
select isValid('blah','number').resulttext from dual; -- returns: Valid formats are: 12345, 12345.67, -12345, etc...
select isValid(1234567890.123,'number') from dual; -- returns: 1,{null}
select isValid(1234567890.123,'number').result from dual; -- returns: 1
select isValid(1234567890.123,'number').resulttext from dual; -- returns: {null}
/* Example Usage in PL/SQL */
declare
temp valObj;
begin
temp := isValid('blah','number');
if (temp.result = 0) then
dbms_output.put_line(temp.resulttext);
else
dbms_output.put_line('Valid');
end if;
end;
/
My questions are:
When using it in PL/SQL I would love to be able to do boolean checks instead like this: if (temp.result) then but I can't figure out a way, cause that won't work in SQL. Should I just add a 3rd boolean attribute to the valObj or is there another way I don't know of?
These validation functions could end up being called within large loops. Knowing that, is this the most efficient way to accomplish these validations?
I'd appreciate any help. Thanks!
UPDATE: I forgot about MEMBER FUNCTIONS. Thanks #Brian McGinity for reminding me. So I'd like to go with this method since it keeps the type and its functions encapsulated together. Would there be any speed difference between this method and a stand-alone function? Would this be compiled and stored the same as a stand-alone function?
create or replace type isValid as object (
result number(1),
resulttext varchar2(32000),
constructor function isValid(v varchar, format varchar) return self as result );
/
create or replace type body isValid as
constructor function isValid(v varchar, format varchar) return self as result as
test number;
begin
if format = 'number' then
begin
test := to_number(v);
self.result := 1;
self.resulttext := null;
return;
exception when VALUE_ERROR then
self.result := 0;
self.resulttext := 'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...';
return;
end;
elsif format = 'phone' then
null; --TO DO
end if;
--and many others...
end;
end;
/
/* Example Usage in SQL */
select isValid('a','number') from dual;
/* Example Usage in PL/SQL */
declare
begin
if (isValid('a','number').result = 1) then
null;
end if;
end;
/
TEST RESULTS:
/* Test isValid (the object member function), this took 7 seconds to run */
declare
begin
for i in 1 .. 2000000 loop
if (isValid('blah','number').result = 1) then
null;
end if;
end loop;
end;
/* Test isValid2 (the stand-alone function), this took 16 seconds to run */
declare
begin
for i in 1 .. 2000000 loop
if (isValid2('blah','number').result = 1) then
null;
end if;
end loop;
end;
Both isValid and isValid2 do the same exact code, they just run this line test := to_number(v); then do the exception if it fails and return the result. Does this appear to be a valid test? The Object member function method is actually faster than a stand-alone function???
The stand-alone function can be much faster if you set it to DETERMINISTIC and if the data is highly repetitive. On my machine this setting decreased run time from 9 seconds to 0.1 seconds. For reasons I don't understand that setting does not improve performance of the object function.
create or replace function isValid2(v in varchar2, format in varchar2)
return valObj
deterministic --<< Hit the turbo button!
is
test number;
begin
if format = 'number' then
begin
test := to_number(v);
return valObj(1,null);
exception when VALUE_ERROR then return valObj(0,'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...');
end;
end if;
end;
/
May also want to consider utilizing pls_integer over number. Don't know if it will buy you much, but documents suggest some gain will be had.
http://docs.oracle.com/cd/B10500_01/appdev.920/a96624/03_types.htm states,
"You use the PLS_INTEGER datatype to store signed integers. Its magnitude range is -2*31 .. 2*31. PLS_INTEGER values require less storage than NUMBER values. Also, PLS_INTEGER operations use machine arithmetic, so they are faster than NUMBER and BINARY_INTEGER operations, which use library arithmetic. For efficiency, use PLS_INTEGER for all calculations that fall within its magnitude range."
A lot of times when we query the database, we just need one column with varchar.
So I've made a nice function for querying the database and putting the results in a stringlist:
function Getdatatostringlist(sqlcomponent, sqlquery: string): TStringlist;
What I'm looking for now is basically the same function but for results with multiple columns where you don't know in advance what type the data is, be it varchar, int, datetime.
What kind of datastructure would be good to use here.
The reason I want this is that I try not to work on open datasets. I like much more to fetch all results into a temporary structure, close the dataset and work on the results.
After Kobiks reply about using in Memory datasets I came up with the following, it's fast put together to test the concept:
procedure TForm1.Button2Click(Sender: TObject);
var
MyDataSet : TAdoDataSet;
begin
MyDataSet := GetDataToDataSet('SELECT naam FROM user WHERE userid = 1', ADOConnection1);
try
Form1.Caption := MyDataSet.FieldByName('naam').AsString;
finally
MyDataSet.free;
end;
end;
function TForm1.GetDataToDataSet(sSql: string; AdoConnection: TADOConnection): TAdoDataSet;
begin
Result := TAdoDataSet.Create(nil);
Result.LockType := ltBatchOptimistic;
Result.Connection := AdoConnection;
Result.CommandText := sSql;
Result.Open;
Result.Connection := nil;
end;
I think this is something to build on.
You should use any disconnected in-memory TDataSet descendant, such as TClientDataSet.
Do not attempt to re-invent the wheel by storing a record-set in some new "Variant" structure. A TClientDataSet already contains all features you need to manipulate a "temporary" data structure.
Here is how you create a TClientDataSet structure:
cds.FieldDefs.Add('id', ftInteger);
cds.FieldDefs.Add('name', ftString, 100);
// ...
// create it
cds.CreateDataSet;
// add some data records
cds.AppendRecord([1, 'Foo']);
cds.AppendRecord([2, 'Bar']);
Many TDataSets has an ability to be used as an in-memory (client) datasets depending on the provider and LockType, for example a TADODataSet with LockType=ltBatchOptimistic could fetch results-set from the server, and then remain disconnected.
For exchanging Data with Excel this structure is usefull, might be useful for other purposes.
Function GetDatasetasDynArray(Ads: TDataset; WithHeader: Boolean = true): Variant;
// 20130118 by Thomas Wassermann
var
i, x, y: Integer;
Fields: Array of Integer;
begin
x := 0;
y := Ads.RecordCount;
if WithHeader then
inc(y);
SetLength(Fields, Ads.FieldCount);
for i := 0 to Ads.FieldCount - 1 do
if Ads.Fields[i].Visible then
begin
Fields[x] := i;
inc(x);
end;
SetLength(Fields, x);
Result := VarArrayCreate([0, y - 1 , 0, length(Fields) - 1], VarVariant);
y := 0;
if WithHeader then
begin
for i := Low(Fields) to High(Fields) do
begin
Result[y, i] := Ads.Fields[Fields[i]].DisplayLabel;
end;
inc(y);
end;
try
Ads.DisableControls;
Ads.First;
while not Ads.EOF do
begin
for i := Low(Fields) to High(Fields) do
begin
Result[y, i] := Ads.Fields[Fields[i]].Value;
end;
Ads.Next;
inc(y);
end;
finally
Ads.EnableControls;
end;
end;
procedure TForm1.Button1Click(Sender: TObject);
var
DynArray:Variant;
begin
DynArray := GetDatasetasDynArray(Adodataset1,true);
//DynArray[0,x] Header or First row
//DynArray[1,x] First row or SecondRow
Excel.Range.Value := DynArray;
end;
Why don't you like working with open datasets? They usually do not block the server. Copying the data from the dataset to whatever you want is extra overhead which is most likely not necessary.
A dataset provides exactly the functionality you want: A matrix with variable columns and rows.
EDIT: However, if you have iterate through the dataset often, you should consider creating a class holding the relevant information and then copy the data into a generic list, dictionary, tree or whatever you need as fast lookup structure.
Of course you could think of building something smart which can be as flexible as a dataset but: The more general things get, the poorer the performance (usually).