I was given a stored proc which generates an open cursor which is passed as output to a reporting tool. I re-wrote this stored proc to improve performance. What I'd like to do is to show that the two result sets are the same for a given set of input parameters.
Something that is the equivalent of:
select * from CURSOR_NEW
minus
select * from CURSOR_OLD
union all
select * from CURSOR_OLD
minus
select * from CURSOR_NEW
Each cursor returns several dozen columns from a large subset of tables. Each row has an id value, and a long list of other column values for that id. I would want to check:
Both cursors are returning the same set of ids (I already checked this)
Both cursors have the same list of values for each id they have in common
If it was just one or two columns, I could concatenate them and find a hash and then sum it up over the cursor. Or another way might be to create a parent program that inserted the cursor results into a global temp table and compared the results. But since it's several dozen columns I'm trying to find a less brute force approach to doing the comparison.
Also it would be nice if the solution was scalable for other situations that involved different cursors, so it wouldn't have to be manually re-written each time, since this is a situation I'm running into more often.
I figured out a way to do this. It was a lot more complicated than I expected. I ended up using some DBMS_SQL procedures that allow converting REFCURSORs to defined cursors. Oracle has documentation on it here:
http://docs.oracle.com/cd/B28359_01/appdev.111/b28370/dynamic.htm#LNPLS00001
After that I concatenated the row values into a string and printed the hash. For bigger cursors, I will change concat_col_vals to use a CLOB to prevent it from overflowing.
p_testCursors returns a simple refcursor for example purposes.
declare
cx_1 sys_refcursor;
c NUMBER;
desctab DBMS_SQL.DESC_TAB;
colcnt NUMBER;
stringvar VARCHAR2(4000);
numvar NUMBER;
datevar DATE;
concat_col_vals varchar2(4000);
col_hash number;
h raw(32767);
n number;
BEGIN
p_testCursors(cx_1);
c := DBMS_SQL.TO_CURSOR_NUMBER(cx_1);
DBMS_SQL.DESCRIBE_COLUMNS(c, colcnt, desctab);
-- Define columns:
FOR i IN 1 .. colcnt LOOP
IF desctab(i).col_type = 2 THEN
DBMS_SQL.DEFINE_COLUMN(c, i, numvar);
ELSIF desctab(i).col_type = 12 THEN
DBMS_SQL.DEFINE_COLUMN(c, i, datevar);
-- statements
ELSE
DBMS_SQL.DEFINE_COLUMN(c, i, stringvar, 4000);
END IF;
END LOOP;
-- Fetch rows with DBMS_SQL package:
WHILE DBMS_SQL.FETCH_ROWS(c) > 0 LOOP
concat_col_vals := '~';
FOR i IN 1 .. colcnt LOOP
IF (desctab(i).col_type = 1) THEN
DBMS_SQL.COLUMN_VALUE(c, i, stringvar);
--Dbms_Output.Put_Line(stringvar);
concat_col_vals := concat_col_vals || '~' || stringvar;
ELSIF (desctab(i).col_type = 2) THEN
DBMS_SQL.COLUMN_VALUE(c, i, numvar);
--Dbms_Output.Put_Line(numvar);
concat_col_vals := concat_col_vals || '~' || to_char(numvar);
ELSIF (desctab(i).col_type = 12) THEN
DBMS_SQL.COLUMN_VALUE(c, i, datevar);
--Dbms_Output.Put_Line(datevar);
concat_col_vals := concat_col_vals || '~' || to_char(datevar);
-- statements
END IF;
END LOOP;
DBMS_OUTPUT.PUT_LINE(concat_col_vals);
col_hash := DBMS_UTILITY.GET_SQL_HASH(concat_col_vals, h, n);
DBMS_OUTPUT.PUT_LINE('Return Value: ' || TO_CHAR(col_hash));
DBMS_OUTPUT.PUT_LINE('Hash: ' || h);
END LOOP;
DBMS_SQL.CLOSE_CURSOR(c);
END;
/
This is not easy task for Oracle.
Very good article you can find on dba-oracle web:
Sql patterns symmetric diff
and Convert set to join sql parameter
If you need it often, you can:
add "hash column" and fill it always with insert using trigger, or
for each table in cursor output get unique value (create unique index) and compare only this column wiht anijoin
and you can find other possibilities in article.
Related
I have a table TestTable with columns of col_test1, col_test2, col_test3 ...
and I want to create a loop that accesses each of these columns individually and find the max value and place it in the variable made in the declare block and simply dbms.out.put it.
Declare
my_array sys.dbms_debug_vc2coll := sys.dbms_debug_vc2coll('col_test1','col_test2','col_test2');
v_test number(8,0);
Begin
for r in my_array.first..my_array.last
loop
select max(my_array(r)) into v_test from TestTable;
dbms_output.put_line(v_test);
end loop;
End;
/
The output I get is just the string 'col_test1'which should be 50.
This is done through oracle SQL. Is there any way to achieve this?
You could use dynamic SQL for this
Declare
my_array sys.dbms_debug_vc2coll := sys.dbms_debug_vc2coll('col_test1','col_test2','col_test2');
v_test number(8,0);
Begin
for r in my_array.first..my_array.last
loop
execute immediate 'select max(' || my_array(r) || ') from TestTable'
into v_test;
dbms_output.put_line(v_test);
end loop;
End;
If you're going to resort to dynamic SQL, however, it would generally make more sense to build a single SQL statement that took that max of all three columns in one pass rather than potentially doing three separate table scans on the same table.
I am parsing string into comma separated and inserting them to global table. The performance is good when inserting around 5k records, performance sucks if the inserting record is around 40k+. The global table has only one column. I thought using bulk fetch and forall will increase the performance, but it’s not the case so far. How can I rewrite below insertion query or any other ways this can be achieved for inserting large records? help will be highly appreciated. I did testing by running insert query by its own and it’s taking long time to process if data size is large.
//large string
emp_refno in CLOB;
CREATE OR replace PROCEDURE employee( emp_refno IN CLOB ) AS
c_limit PLS_INTEGER := 1000;
CURSOR token_cur IS
WITH inputs(str) AS
( SELECT to_clob(emp_refno)
FROM dual ),
prep(s,n,token,st_pos,end_pos ) AS
(
SELECT ','|| str || ',',-1,NULL,NULL,1
FROM inputs
UNION ALL
SELECT s, n + 1,substr(s, st_pos, end_pos - st_pos),
end_pos + 1,instr(s, ',', 1, n + 3)
FROM prep
WHERE end_pos != 0
)
SELECT token
FROM prep
WHERE n > 0;
TYPE token_t
IS
TABLE OF CLOB;
rec_token_t TOKEN_T;
BEGIN
OPEN token_cur;
LOOP
FETCH token_cur bulk collect
INTO rec_token_t limit c_limit;
IF rec_token_t.count > 0 THEN
forall rec IN rec_token_t.first ..rec_token_t.last
INSERT INTO globaltemp_emp
VALUES ( rec_token_t(rec) );
COMMIT;
END IF;
EXIT
WHEN rec_token_t.count = 0;
END LOOP;
OPEN p_resultset FOR
SELECT e.empname,
e.empaddress,
f.department
FROM employee e
join department f
ON e.emp_id = t.emp_id
AND e.emp_refno IN
(
SELECT emp_refno
FROM globaltemp_emp) //USING gtt IN subquery
END;
I have adapted a function which gives better performance.For 90k records, it returns in 13 seconds.Also reduce the c_limit to 250
You can adapt the below
CREATE OR replace FUNCTION pipe_clob ( p_clob IN CLOB,
p_max_lengthb IN INTEGER DEFAULT 4000,
p_rec_delim IN VARCHAR2 DEFAULT '
' )
RETURN sys.odcivarchar2list pipelined authid current_user AS
/*
Break CLOB into VARCHAR2 sized bites.
Reduce p_max_lengthb if you need to expand the VARCHAR2
in later processing.
Last record delimiter in each bite is not returned,
but if it is a newline and the output is spooled
the newline will come back in the spooled output.
Note: this cannot work if the CLOB contains more than
<p_max_lengthb> consecutive bytes without a record delimiter.
*/
l_amount INTEGER;
l_offset INTEGER;
l_buffer VARCHAR2(32767 byte);
l_out VARCHAR2(32767 byte);
l_buff_lengthb INTEGER;
l_occurence INTEGER;
l_rec_delim_length INTEGER := length(p_rec_delim);
l_max_length INTEGER;
l_prev_length INTEGER;
BEGIN
IF p_max_lengthb > 4000 THEN
raise_application_error(-20001, 'Maximum record length (p_max_lengthb) cannot be greater than 4000.');
ELSIF p_max_lengthb < 10 THEN
raise_application_error(-20002, 'Maximum record length (p_max_lengthb) cannot be less than 10.');
END IF;
IF p_rec_delim IS NULL THEN
raise_application_error(-20003, 'Record delimiter (p_rec_delim) cannot be null.');
END IF;
/* This version is limited to 4000 byte output, so I can afford to ask for 4001
in case the record is exactly 4000 bytes long.
*/
l_max_length:=dbms_lob.instr(p_clob,p_rec_delim,1,1)-1;
l_prev_length:=0;
l_amount := l_max_length + l_rec_delim_length;
l_offset := 1;
WHILE (l_amount = l_max_length + l_rec_delim_length
AND
l_amount > 0)
LOOP
BEGIN
dbms_lob.READ ( p_clob, l_amount, l_offset, l_buffer );
EXCEPTION
WHEN no_data_found THEN
l_amount := 0;
END;
IF l_amount = 0 THEN
EXIT;
ELSIF lengthb(l_buffer) <= l_max_length THEN
pipe ROW(rtrim(l_buffer, p_rec_delim));
EXIT;
END IF;
l_buff_lengthb := l_max_length + l_rec_delim_length;
l_occurence := 0;
WHILE l_buff_lengthb > l_max_length
LOOP
l_occurence := l_occurence + 1;
l_buff_lengthb := instrb(l_buffer,p_rec_delim, -1, l_occurence) - 1;
END LOOP;
IF l_buff_lengthb < 0 THEN
IF l_amount = l_max_length + l_rec_delim_length THEN
raise_application_error( -20004, 'Input clob at offset '
||l_offset
||' for lengthb '
||l_max_length
||' has no record delimiter' );
END IF;
END IF;
l_out := substrb(l_buffer, 1, l_buff_lengthb);
pipe ROW(l_out);
l_prev_length:=dbms_lob.instr(p_clob,p_rec_delim,l_offset,1)-1;--san temp
l_offset := l_offset + nvl(length(l_out),0) + l_rec_delim_length;
l_max_length:=dbms_lob.instr(p_clob,p_rec_delim,l_offset,1)-1;--san temp
l_max_length:=l_max_length-l_prev_length;
l_amount := l_max_length + l_rec_delim_length;
END LOOP;
RETURN;
END;
and then use like the below in the cursor in your procedure
CURSOR token_cur IS
select * from table (pipe_clob(emp_refno||',',10,','));
Three quick suggestions:
Perform commit for around 1000(or in batches) records rather than doing for each.
Replace in with exists for the Ref cursor.
Index globaltemp_emp.emp_refno if it doesn't have already.
Additionally recommend to run explain plan for each of the DML operation to check for any odd behaviour.
user uploads text file and I parse that text file as a comma seperated string and pass it to Oracle DB.
You are doing a bunch of work to turn that file into a string and then another bunch of work to convert that string into a table. As many people have observed before me, the best performance comes from not doing work we don't have to do.
In this case this means you should load the file's contents directly into the database. We can do this with an external table. This is a mechanism which allows us to query data from a file on the server using SQL. It would look something like this:
create table emp_refno_load
(emp_refno varchar2(24))
organization external
(type oracle_loader
default directory file_upload_dir
access parameters
(records delimited by newline
fields (employee_number char(24)
)
)
location ('some_file.txt')
);
Then you can discard your stored procedure and temporary table and re-write your query to something like this:
SELECT e.empname,
e.empaddress,
f.department
FROM emp_refno_load l
join employee e ON l.emp_refno = e.emp_refno
join department f ON e.emp_id = f.emp_id
The one snag with external tables is they require access to an OS directory (file_upload_dir in my example above) and some database security policies are weird about that. However the performance benefits and simplicity of approach should carry the day.
Find out more.
An external table is undoubtedly the most performative approach (until you hit millions of roads and then you need SQL*Loader ).
I am familiar with MSSQL and using a parameter within the query, but I am not sure how I would do this within PL/SQL.
DECLARE
LSITEID NUMBER := 100001;
BEGIN
SELECT * from invoicehead ih
JOIN sitemaster sm on sm.SITEIID = ih.SITEIID
JOIN invoiceline il on il.invoiceIID = ih.invoiceIID
WHERE
ih.StartDate BETWEEN '2015-12-01' AND '2016-03-07'
AND SITEIID IN ( LSITEID)
END;
Right now I am testing this within Pl/SQL. But essentially I would be passing in the query with the parameter from MSSQL Linked Server OPENQuery.
How I can run the above query in PL/SQL with the parameter?
There is plenty of other resource for finding an answer, e.g. here (Tutorialspoint) or specifically here (plsql-tutorial). But perhaps I have missed your point.
To not remain on merely citing links, your query could look like this:
DECLARE
LSITEID integer;
BEGIN
LSITEID := 100001;
-- dostuff
END;
Two things to note: First, in a declare part (as I have learnt it) you should avoid assigning values. Second, if you intend to pass in different parameters you could/should use a procedure.
In PL/SQL you just use the name of the argument. In the following example, the argument is P_VALUE, note select statement says where dummy = p_value.
DECLARE
FUNCTION dummycount (p_value IN DUAL.dummy%TYPE)
RETURN INTEGER
AS
l_ret INTEGER;
BEGIN
SELECT COUNT (*) c
INTO l_ret
FROM DUAL
WHERE dummy = p_value;
RETURN l_ret;
END dummycount;
BEGIN
DBMS_OUTPUT.put_line ('A: ' || dummycount (p_value => 'A'));
DBMS_OUTPUT.put_line ('X: ' || dummycount (p_value => 'X'));
END;
This results in the following output:
A: 0
X: 1
I have the following problem. I am an experienced Java programmer but am a bit of a n00b at SQL and PL/SQL.
I need to do the following.
1 Pass in a few arrays and some other variables into a procedure
2 Loop on the values in the arrays (they all have the same number of items) and dynamically create an SQL statement
3 Run this statement and add it to the result set (which is an OUT parameter of the procedure)
I already have experience of creating an SQL query on the fly, running it and adding the result to a result set (which is a REF CURSOR) but I'm not sure how I'd loop and add the results of each call to the query to the same result set. I'm not even sure if this is possible.
Here's what I have so far (code edited for simplicity). I know it's wrong because I'm just replacing the contents of the RESULT_SET with the most recent query result (and this is being confirmed in the Java which is calling this procedure).
Any and all help would be greatly appreciated.
TYPE REF_CURSOR IS REF CURSOR;
PROCEDURE GET_DATA_FASTER(in_seq_numbers IN seq_numbers_array, in_values IN text_array, in_items IN text_array, list IN VARCHAR2, RESULT_SET OUT REF_CURSOR) AS
query_str VARCHAR2(4000);
seq_number NUMBER;
the_value VARCHAR2(10);
the_item VARCHAR2(10);
BEGIN
FOR i IN 1..in_seq_numbers.COUNT
LOOP
seq_number := in_seq_numbers(i);
the_value := trim(in_values(i));
the_item := trim(in_items(i));
query_str := 'SELECT distinct '||seq_number||' as seq, value, item
FROM my_table ai';
query_str := query_str || '
WHERE ai.value = '''||the_value||''' AND ai.item = '''||the_item||'''
AND ai.param = ''BOOK''
AND ai.prod in (' || list || ');
OPEN RESULT_SET FOR query_str;
END LOOP;
EXCEPTION WHEN OTHERS THEN
RAISE;
END GET_DATA_FASTER;
A pipelined table function seems a better fit for what you want, especially if all you're doing is retrieving data. See http://www.oracle-base.com/articles/misc/pipelined-table-functions.php
What you do is create a type for your output row. So in your case you would create an object such as
CREATE TYPE get_data_faster_row AS OBJECT(
seq NUMBER(15,2),
value VARCHAR2(10),
item VARCHAR2(10)
);
Then create a table type which is a table made up of your row type above
CREATE TYPE get_data_faster_data IS TABLE OF get_data_faster_row;
Then create your table function that returns the data in a pipelined manner. Pipelined in Oracle is a bit like a yield return in .net (not sure if you're familiar with that). You find all of the rows that you want and "pipe" them out one at a time in a loop. When your function completes the table that's returned consists of all the rows you piped out.
CREATE FUNCTION Get_Data_Faster(params) RETURN get_data_faster_data PIPELINED AS
BEGIN
-- Iterate through your parameters
--Iterate through the results of the select using
-- the current parameters. You'll probably need a
-- cursor for this
PIPE ROW(get_data_faster_row(seq, value, item));
LOOP;
LOOP;
END;
EDIT: Following Alex's comment below, you need something like this. I haven't been able to test this but it should get you started:
CREATE FUNCTION Get_Data_Faster(in_seq_numbers IN seq_numbers_array, in_values IN text_array, in_items IN text_array, list IN VARCHAR2) RETURN get_data_faster_data PIPELINED AS
TYPE r_cursor IS REF CURSOR;
query_results r_cursor;
results_out get_data_faster_row := get_data_faster_row(NULL, NULL, NULL);
query_str VARCHAR2(4000);
seq_number NUMBER;
the_value VARCHAR2(10);
the_item VARCHAR2(10);
BEGIN
FOR i IN 1..in_seq_number.COUNT
LOOP
seq_number := in_seq_numbers(i);
the_value := trim(in_values(i));
the_item := trim(in_items(i));
query_str := 'SELECT distinct '||seq_number||' as seq, value, item
FROM my_table ai';
query_str := query_str || '
WHERE ai.value = '''||the_value||''' AND ai.item = '''||the_item||'''
AND ai.param = ''BOOK''
AND ai.prod in (' || list || ');
OPEN query_results FOR query_str;
LOOP
FETCH query_results INTO
results_out.seq,
results_out.value,
results_out.item;
EXIT WHEN query_results%NOTFOUND;
PIPE ROW(results_out);
END LOOP;
CLOSE query_results;
END LOOP;
END;
Extra info from Alex's comment below useful for the answer:
you can have multiple loops from different sources, and as long as the
data from each be put into the same object type, you can just keep
pumping them out with pipe row statements anywhere in the function.
The caller sees them as a table with the rows in the order you pipe
them. Rather than call a procedure and get a result set as an output
parameter, you can query as select seq, value, item from
table(package.get_data_faster(a, b, c, d)), and of course you can
still have an order by clause if the order they're piped isn't what
you want.
I am getting a ORA-06532 error in my pl/sql procedure. It has to do with my array, and it seems to be happening on the line that starts: "term_1 := ...". The query that selects into gets 7 records, so it should be the same amount as what I am calling for. I am a little new to the SQL array thing, so I may have missed something obvious. Any help is appreciated, thanks.
DECLARE
listOfTerms VC50_ARRAY;
term_1 varchar2(30);
term_2 varchar2(30);
term_3 varchar2(30);
term_4 varchar2(30);
term_5 varchar2(30);
term_6 varchar2(30);
term_7 varchar2(30);
BEGIN
SELECT apl.awdp_acad_terms
BULK COLLECT INTO listOfTerms
FROM fa_years fay
JOIN award_periods_ls apl
ON apl.award_periods_id = fay.award_periods_id
WHERE (SELECT b.awdp_fa_year as faYear
FROM award_periods_ls a
JOIN coll18_test.fa_years b ON a.award_periods_id = b.award_periods_id
WHERE awdp_acad_terms = v_ug_term) = fay.awdp_fa_year
ORDER BY apl.awdp_acad_terms DESC;
term_1 := listOFTerms(1);
term_2 := listOFTerms(2);
term_3 := listOFTerms(3);
term_4 := listOFTerms(4);
term_5 := listOFTerms(5);
term_6 := listOFTerms(6);
term_7 := listOFTerms(7);
I think you're over-complicating this. You code is equivalent to:
DECLARE
-- Declare the cursor explicitly.
cursor c_my_cursor is
SELECT apl.awdp_acad_terms
FROM fa_years fay
JOIN award_periods_ls apl
ON apl.award_periods_id = fay.award_periods_id
WHERE ( SELECT b.awdp_fa_year as faYear
FROM award_periods_ls a
JOIN coll18_test.fa_years b
ON a.award_periods_id = b.award_periods_id
WHERE awdp_acad_terms = v_ug_term ) = fay.awdp_fa_year
ORDER BY apl.awdp_acad_terms DESC;
-- Create a-user defined type that is the same as a single row in the cursor.
type t__listOfTerms is table of c_my_cursor%rowtype index by binary_integer;
-- Initialise a variable that is of data-type t__listofterms.
t_listofterms t__listofterms;
BEGIN
open c_my_cursor;
fetch c_my_cursor bulk collect into t_listofterms;
close c_my_cursor;
END;
You can then reference the items in your type by their index values so term_1 is the same as t_listofterms(1). There's no need create an additional variable with the same value; you can reference it in the same way so length(term1) and length(t_listofterms(1)) are also the same.
There's a lot of stuff out there about array processing but PSOUG is helpful as is the documentation.
Judging by your comment you may be referencing the collection explicitly, i.e. something := t_listofterms(7). This assumes that there is a specific number of rows. Bulk collect fills a collection from 1 to n, where n is the number of rows returned by the query. It's often better to loop through this if you want to do something with it rather than explicit referencing. Something like,
for i in t_listofterms.first .. t_listofterms.last loop
do_something;
end loop;