oracle inline function called multiple times - sql

When calling a function via an inline select statement, when the function is returning a custom type, Oracle seems to execute the function equal to the number of arguments +1. This seems to happen when the select is included as a CTAS or an insert/select.
Has anyone seen this before? Is this an Oracle bug? I would expect the function to be called once per row in the table.
--Inline function gets called for the number of arguments +1
--drop table t
create table t(
id number,
l_blob blob
);
insert into t values(1, utl_raw.cast_to_raw('SampleString'));
COMMIT;
create table tmp_ts (c1 timestamp);
create or replace type test_type as object(
c1 varchar2(32)
,c2 varchar2(32)
);
/
create or replace FUNCTION test_function (p_blob blob, p_date date)
RETURN test_type
IS
BEGIN
--This could also be a DBMS_OUTPUT.PUT_LINE statement
insert into tmp_ts VALUES (systimestamp);
return test_type(null,null);
END test_function;
/
--0
select count(*) from tmp_ts;
--Call function on 1 row table - function should just insert 1 row into tmp_ts
create table tst_table as
select test_function(l_blob, '25-JAN-09') as c1
from t;
--it actually inserts 3
select count(*) from tmp_ts;
Example where increasing the argument call for the type increases the number of time the function is executed
--Same example with more arguements - 6 arguements here
--Inline function gets called for the number of arguments +1
--drop table t
create table t2(
id number,
l_blob blob
);
insert into t2 values(1, utl_raw.cast_to_raw('SampleString'));
COMMIT;
create table tmp_ts2 (c1 timestamp);
create or replace type test_type2 as object(
c1 varchar2(32)
,c2 varchar2(32)
,c3 varchar2(32)
,c4 varchar2(32)
,c5 varchar2(32)
,c6 varchar2(32)
);
/
create or replace FUNCTION test_function2 (p_blob blob, p_date date)
RETURN test_type2
IS
BEGIN
insert into tmp_ts2 VALUES (systimestamp);
return test_type2(null,null,null,null,null,null);
END test_function2;
/
--0
select count(*) from tmp_ts2;
--Call function on 1 row table - function should just insert 1 row into tmp_ts
create table tst_table2 as
select test_function2(l_blob, '25-JAN-09') as c1
from t;
--it actually inserts 7
select count(*) from tmp_ts2;
Any help/feedback is greatly appreciated.

First: It is a bug that you can even perform a DML inside a function which is called in a SELECT Statement. This should raise an exception.
Otherwise Oracle makes absolutely no guarantee how often Functions in a SQL-Select are executed, it could be once per row, ten times per row or just once for the whole query (with caching) - so however often it is called, this conforms to the specifications.
In this special case it will call the function for each attribute of the returning type, since oracle will not insert the object type as one memory-structure, but use the function like a table with multiple columns and read each column individually like this:
INSERT VALUES ( myFunc(x).attribute1, myFunc(x).attribute2 );
The important part: Never make any assumptions about how often a FUNCTION is called when you use it in an SQL Statement!!! At any time the function could be called again by the optimizer, maybe for sampling or caching...
Preferred solution: Pipelined Functions - a pipelined Function can be called like a Table and will only be called once. You can pass in a cursor which the function uses for input processing and do the whole data-transformation and logging and everything in the function.

Related

Why is RETURN QUERY returning a string instead of a TABLE

This MWE is NOT how you would typically solve this problem, however, it is as simple as I can explain the problem I am encountering. I am merely trying to point out 2 things
I am doing more than simply returning the contents of a Table
What is being returned is NOT being returned as a Table but a String
Supporting SQL Statements:
DROP DATABASE IF EXISTS test;
CREATE DATABASE test;
\c test
CREATE TABLE credit_card(
id BIGSERIAL PRIMARY KEY,
balance BIGINT
);
Functions:
CREATE FUNCTION get_credit_card(
p_id BIGINT
)
RETURNS TABLE(
id BIGINT,
balance BIGINT
)
AS $$
DECLARE
BEGIN
RETURN QUERY
SELECT
credit_card.id,
credit_card.balance
FROM
credit_card
WHERE
credit_card.id = p_id;
END $$ LAnguage 'plpgsql';
CREATE FUNCTION pay_with_card(
p_id BIGINT,
p_amount BIGINT
)
RETURNS TABLE(
id BIGINT,
balance BIGINT
)
AS $$
DECLARE
v_balance BIGINT;
BEGIN
SELECT
credit_card.balance
FROM
credit_card
INTO
v_balance
WHERE
credit_card.id = p_id;
IF v_balance < p_amount
THEN
RETURN;
END IF;
UPDATE
credit_card
SET
balance = credit_card.balance - p_amount;
RETURN QUERY
SELECT get_credit_card (p_id);
END $$ LAnguage 'plpgsql';
Populate Table and Call function:
INSERT INTO credit_card
(balance)
VALUES
(100);
SELECT
pay_with_card (1, 100);
Error:
DROP DATABASE
CREATE DATABASE
You are now connected to database "test" as user "postgres".
CREATE TABLE
CREATE FUNCTION
CREATE FUNCTION
INSERT 0 1
psql:test.sql:74: ERROR: structure of query does not match function result type
DETAIL: Returned type record does not match expected type bigint in column 1.
CONTEXT: PL/pgSQL function pay_with_card(bigint,bigint) line 24 at RETURN QUERY
It took me a long time to figure out that pay_with_card is returning a String, or what appears to be a String, instead of a TABLE(id BIGINT, balance BIGINT). With the Python psycopg2 library, the returned query is
[('(1,100)'),]
So my entire code is breaking because I can't get the values (unless I hack it and use string manipulation.
Question:
How can I fix it so that it returns the correct query like so
[(1,100),]
An alternative to the hint in horse_with_no_name's comment, you can replace
RETURN QUERY
SELECT get_credit_card (p_id);
with
RETURN QUERY SELECT (get_credit_card(p_id)).*;
You need some way of expanding the returned record back into its constituent fields. (I think horse’s SELECT * … has the same effect.)

Is SELECT "faster" than function with nested INSERT?

I'm using a function that inserts a row to a table if it doesn't exist, then returns the id of the row.
Whenever I put the function inside a SELECT statement, with values that don't exist in the table yet, e.g.:
SELECT * FROM table WHERE id = function(123);
... it returns an empty row. However, running it again with the same values will return the row with the values I want to see.
Why does this happen? Is the INSERT running behind the SELECT speed? Or does PostgreSQL cache the table when it didn't exist, and at next run, it displays the result?
Here's a ready to use example of how this issue can occur:
CREATE TABLE IF NOT EXISTS test_table(
id INTEGER,
tvalue boolean
);
CREATE OR REPLACE FUNCTION test_function(user_id INTEGER)
RETURNS integer
LANGUAGE 'plpgsql'
AS $$
DECLARE
__user_id INTEGER;
BEGIN
EXECUTE format('SELECT * FROM test_table WHERE id = $1')
USING user_id
INTO __user_id;
IF __user_id IS NOT NULL THEN
RETURN __user_id;
ELSE
INSERT INTO test_table(id, tvalue)
VALUES (user_id, TRUE)
RETURNING id
INTO __user_id;
RETURN __user_id;
END IF;
END;
$$;
Call:
SELECT * FROM test_table WHERE id = test_function(4);
To reproduce the issue, pass any integer that doesn't exist in the table, yet.
The example is broken in multiple places.
No need for dynamic SQL with EXECUTE.
SELECT * in the function is wrong.
Your table definition should have a UNIQUE or PRIMARY KEY constraint on (id).
Most importantly, the final SELECT statement is bound to fail. Since the function is VOLATILE (has to be), it is evaluated once for every existing row in the table. Even if that worked, it would be a performance nightmare. But it does not. Like #user2864740 commented, there is also a problem with visibility. Postgres checks every existing row against the result of the function, which in turn adds 1 or more rows, and those rows are not yet in the snapshot the SELECT is operating on.
SELECT * FROM test_table WHERE id = test_function(4);
This would work (but see below!):
CREATE TABLE test_table (
id int PRIMARY KEY --!
, tvalue bool
);
CREATE OR REPLACE FUNCTION test_function(_user_id int)
RETURNS test_table LANGUAGE sql AS
$func$
WITH ins AS (
INSERT INTO test_table(id, tvalue)
VALUES (_user_id, TRUE)
ON CONFLICT DO NOTHING
RETURNING *
)
TABLE ins
UNION ALL
SELECT * FROM test_table WHERE id = _user_id
LIMIT 1
$func$;
And replace your SELECT with just:
SELECT * FROM test_function(1);
db<>fiddle here
Related:
Return a value if no record is found
How to use RETURNING with ON CONFLICT in PostgreSQL?
There is still a race condition for concurrent calls. If that can happen, consider:
Is SELECT or INSERT in a function prone to race conditions?

PostgreSQL - Shared temp table between functions

I wnat to know if its possible to share a temporary table between functions that are called in a "main function", like this:
-- some sub function
create or replace function up_sub_function (str text)
returns table (id int, descr text) as $$
begin
return query select * from temp_table where descr like concat('%', str , '%');
end; $$
language plpgsql;
-- main function
create or replace function up_main_function ()
returns table (id int, descr text) as $$
begin
create temporary table temp_table if not exists (
id int,
descr text
);
insert into temp_campaigns select id, descr from test_table;
return query select * from up_sub_function('a');
end; $$
language plpgsql;
BEGIN;
select * from up_main_function();
drop table temp_table;
COMMIT;
If you can show me the correct way to achieve this, I want to be able to populate a temporary table and then filter rows by calling othe functions inside the main function.
Thanks ans happy programming! :)
See the documentation https://www.postgresql.org/docs/current/static/sql-createtable.html
temp tables are valid for the entire session. That is as long as you stay connected to the database.
In your case you only need it during the transaction. So you should create it with ON COMMIT DROP
create temporary table temp_table if not exists (
id int,
descr text
) ON COMMIT DROP;
Once you created the table you can use it within any function in the current transaction.
You do not need the BEGIN to start the transaction. A transaction is automatically started when the outer function is called.
Nested function calls share the same transaction. So they all see the table.

How to use in statement with nested table

Hey there I have a function, and part of the function is to make sure that the selected value is within the passed in table of varchar2s. To start I declare a varchar2 table type like so.
create or replace type Varchar2Table is table of varchar2(200)
Then I have the function which accepts the nested table parameter and has a select statement on them.
function SelectPeople(inputNames Varchar2Table) return People
begin
--stuff
select * from person_table where name in inputNames; --line of interest
--more stuff
end;
This doesn't seem to work though, I get the following error:
ORA-00932: inconsistent datatypes: expected NUMBER got
ENGSPL5.VARCHAR2TABLE
Any suggestions?
The TABLE operator allows nested tables to be used in SQL statements. The function was also missing an IS and an INTO.
create or replace type Varchar2Table is table of varchar2(200);
create table person_table(id number, name varchar2(100));
create or replace function SelectPeople(inputNames Varchar2Table) return number
is --Missing "IS".
type numberTable is table of number; --Need a collection to store results.
numbers numberTable;
begin
select id
bulk collect into numbers --Missing "INTO".
from person_table
where name in (select column_value from table(inputNames)); --Missing "TABLE".
--Alternatively a multiset condition can be used.
--where name member of inputNames;
--Dummy return value to make the function compile.
return 1;
end;
/

How to find the source table rows used by a big complex query

We have a huge Oracle SQL query in our project which is using many views and tables for source data.
Is there any way to find the list of rows fetched from each source table by this big query when I run it?
Basically what we are trying to do is to create the bare minimum number of rows in the source tables so that the outer most big query returns at least a single record.
I have tried to run the smaller queries individually. But it is really time consuming and tedious. So I was wondering if there is a smarter way of doing this.
You can use Fine Grained Access Control. Here is how you might do it:
Step 1: Create a table to hold the list of rowids (i.e., the results you are looking for in this exercise)
CREATE TABLE matt_selected_rowids ( row_id rowid );
Step 2: Create a FGAC handler that will add a predicate whenever a base table is selected.
CREATE OR REPLACE PACKAGE matt_fgac_handler IS
FUNCTION record_rowid ( p_rowid rowid ) RETURN NUMBER DETERMINISTIC;
FUNCTION add_rowid_predicate (d1 varchar2, d2 varchar2 ) RETURN VARCHAR2;
END matt_fgac_handler;
CREATE OR REPLACE PACKAGE BODY matt_fgac_handler IS
FUNCTION record_rowid ( p_rowid rowid ) RETURN NUMBER DETERMINISTIC IS
PRAGMA AUTONOMOUS_TRANSACTION;
BEGIN
INSERT INTO matt_selected_rowids( row_id ) values ( p_rowid );
COMMIT;
RETURN -1;
END record_rowid;
FUNCTION add_rowid_predicate (d1 varchar2, d2 varchar2 ) RETURN VARCHAR2 IS
BEGIN
RETURN 'matt_fgac_handler.record_rowid (rowid) = -1';
END add_rowid_predicate;
END matt_fgac_handler;
Step 3: Add a policy to each base table used by your view (you can get the list by using DBA_DEPENDENCIES recursively, or just doing an explain plan and eyeballing it).
E.g.,
CREATE TABLE matt_table ( a number, b varchar2(80) );
CREATE INDEX matt_table_n1 on matt_table(a);
insert into matt_table values (1,'A');
insert into matt_table values (2,'B');
insert into matt_table values (3,'C');
insert into matt_table values (3,'D');
insert into matt_table values (3,'E');
insert into matt_table values (3,'F');
insert into matt_table values (4,'G');
insert into matt_table values (4,'H');
COMMIT;
BEGIN
DBMS_RLS.ADD_POLICY('APPS','MATT_TABLE','record_rowids_policy', NULL, 'matt_fgac_handler.add_rowid_predicate', 'select');
END;
So, at this point, whenever a user selects from my table, Oracle is automatically going to add a condition that calls my record_rowid function for each rowid.
For example:
delete from matt_selected_rowids;
SELECT /*+ INDEX */ * FROM matt_table where a = 2;
-- This gives your the rowids selected...
select r.row_id, o.object_name from matt_selected_rowids r left join dba_objects o on o.object_id =dbms_rowid.rowid_object(row_id);