PostgreSQL how to disable plan caching to fix my base62 hashing? - sql

From the documentation i gather, that PostgreSQL somehow employs its own cache and prepares satements ahead...
This is really bad for my base62 hash values. At some point, after 2-3 tries, they start returning the same number:
LOG: base62_id.val 1501145675089
CONTEXT: PL/pgSQL function copy_article(text) line 23 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: copied_article_id QQZCFzm | article_count 1
CONTEXT: PL/pgSQL function copy_article(text) line 37 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: base62_id.val 1501145675089
CONTEXT: PL/pgSQL function copy_article(text) line 23 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: copied_article_id QQZCFzm | article_count 1
CONTEXT: PL/pgSQL function copy_article(text) line 37 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: base62_id.val 1501145675089
CONTEXT: PL/pgSQL function copy_article(text) line 23 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: copied_article_id QQZCFzm | article_count 1
CONTEXT: PL/pgSQL function copy_article(text) line 37 at RAISE
STATEMENT: select to_json("public"."copy_article"($1)) as value
LOG: base62_id.val 1501145675089
Here is my function:
CREATE OR REPLACE FUNCTION base62_id() RETURNS character varying
LANGUAGE plpgsql IMMUTABLE
AS $$
DECLARE
chars char[];
ret varchar;
val bigint;
BEGIN
chars := ARRAY['0','1','2','3','4','5','6','7','8','9'
,'A','B','C','D','E','F','G','H','I','J','K','L','M'
,'N','O','P','Q','R','S','T','U','V','W','X','Y','Z'
,'a','b','c','d','e','f','g','h','i','j','k','l','m'
,'n','o','p','q','r','s','t','u','v','w','x','y','z'];
val := (CEIL(EXTRACT(EPOCH FROM now()) * 1000))::bigint;
RAISE LOG 'base62_id.val %', val;
ret := '';
IF val < 0 THEN
val := val * -1;
END IF;
WHILE val != 0 LOOP
ret := chars[(val % 62)+1] || ret;
val := val / 62;
END LOOP;
RETURN ret;
END;$$;
In theory, this should work...
Any ideas?
Edit: How i use the function:
DECLARE
copied_article_id text := base62_id();
duplication_check int := 1;
copied_article articles;
BEGIN
WHILE duplication_check IS NOT NULL LOOP
SELECT COUNT(*) INTO duplication_check FROM articles WHERE id = copied_article_id;
END LOOP;
INSERT ... INTO ... SELECT ...
FROM
articles
WHERE
id = base_id;
SELECT * INTO copied_article FROM articles WHERE id = copied_article_id LIMIT 1;

Your explanation is pretty bogus. Plan caching has nothing to do with this outcome, the plancache doesn't care about individual function outputs.
There's at least two giant gaping bugs in the function:
You declare it IMMUTABLE but you call now() which is not immutable. IMMUTABLE functions must return the same result for every call with the same inputs. In fact, your function must be declared VOLATILE if it's allowed to return a different value for each call with the same inputs.
now() is STABLE. It actually returns the same value for each call within a transaction. So presumably using it when you want unique values makes no sense at all. I imagine you actually want clock_timestamp().
The latter problem with now() (a.k.a. current_timestamp) being STABLE across a transaction is likely the cause for the results you report.
BTW, the function will also probably be amazingly slow implemented in plpgsql. If you can turn it into a set operation plus string_agg it might be more tolerable, but still slow.

Related

PostgreSQL 11 - using format to assign to variable

I have been awake for well beyond my schedule and I have been stuck with this issue for a long time, I don't even know what I am looking for to solve, but I wish to use format to insert values that I'll be using for column names, and then executing it... but it keeps giving me errors no matter how much I try changing it :c
Heres the part that im trying to do something that doesnt work, but i think you get the idea what im trying to achieve
ratelimit := EXECUTE format('(SELECT %I
FROM users.ratelimits
WHERE user_id = $2)
', $1);
and heres the full code for the brave
CREATE OR REPLACE FUNCTION users.consume_ratelimit(_name text,__user_id integer)
RETURNS boolean
LANGUAGE 'plpgsql'
VOLATILE
PARALLEL UNSAFE
COST 100
AS $BODY$DECLARE
ratelimit INTEGER;
reset_timeout timestamptz;
premium BOOLEAN;
BEGIN
ratelimit := EXECUTE format('(SELECT %I
FROM users.ratelimits
WHERE user_id = $2)
', $1);
reset_timeout := EXECUTE format('(SELECT %I_refresh
FROM users.ratelimits
WHERE user_id = $2)
', $1);
premium := (SELECT users.is_premium($2));
IF premium THEN
RETURN TRUE;
ELSIF reset_timeout <= NOW() THEN
UPDATE users.ratelimits
SET image_refresh = NOW() + '1 hour'::interval,
image = DEFAULT
WHERE user_id = $2;
RAISE NOTICE 'reset';
RETURN TRUE;
ELSE
IF ratelimit > 0 THEN
EXECUTE format('UPDATE users.ratelimits
SET %I = %I - 1
WHERE user_id = $2', $1, $1);
RAISE NOTICE 'decrement';
RETURN TRUE;
ELSIF ratelimit <= 0 THEN
RAISE NOTICE 'out of credits';
RETURN FALSE;
ELSE
EXECUTE format('INSERT INTO users.ratelimits(user_id) VALUES ($2)
ON CONFLICT DO UPDATE SET
%I = excluded.%I,
%I_refresh = excluded.%I_refresh', $1, $1, $1, $1);
RAISE NOTICE 'create';
RETURN TRUE;
END IF;
END IF;
END;$BODY$;
As documented in the manual you need to use into together with EXECUTE to store the result into a variable. This can handle multiple columns/variables as well, so you only need a single EXECUTE to get both values.
For clarity you should reference parameters by name, not by position.
EXECUTE format('SELECT %I, %I_refresh
FROM users.ratelimits WHERE user_id = $1'),
_name, _name)
USING __user_id
INTO ratelimit, reset_timeout;
Note the $1 inside the string for format() is a parameter placeholder used when the SQL statement is executed, and will be replaced with the value of the variable specified in the USING clause.
Variable assignment is also more efficient without a SELECT:
premium := users.is_premium(__user_id);
It seems some_var := EXECUTE ... will not work. Additionally, params inside of an EXECUTE statement are not the same as those in the function - you need to supply them to the execute statement.
I have not tested this, but perhaps
-- q declared above as text
q := format('
SELECT %I
FROM users.ratelimits
WHERE user_id = %s;
', $1, $2);
EXECUTE q INTO ratelimit;
will work? I also removed the parens from the query, those are unnecessary and may the problem themselves.
I have tested the function
CREATE OR REPLACE FUNCTION test_sum (a int, b int)
RETURNS int
AS $$
DECLARE
q text;
result int;
BEGIN
q := FORMAT ('SELECT %s + %s', $1, $2);
EXECUTE q INTO result;
RETURN result;
END
$$ LANGUAGE plpgsql
and that does work. If my suggestion above does not work, perhaps you can use the above as a starting point.

Compilation error Oracle SQL

I am trying to learn how to create functions on Oracle. But I when I create my simple function it says that it was create but with compilation error.
This is my code:
create or replace function show_speed
(in_nplate varchar2)
return number
is
ret number(4);
begin
select speed into ret from cars where in_nplate = nplate;
return ret;
end show_speed;
Now I what do this query,
select nplate, speed, show_speed(nplate) from cars;
It has to show all the plates of all cars and their speed 2 times, one for the attribute and second from the function.
I get this error:
exact fetch returns more than requested number of rows. T
he error it show in the line: "select speed into...."
I was looking on internet but I don't find my mistake.
The error text
exact fetch returns more than requested number of rows.
Told you that your table cars has two or more duplicates line.
For example: imagin that you table contain the follow data.
car_____________speed_______in_plate
toyota | 70 | plate_one
shevrolette | 60 | plate_two
volvo | 80 | plate_two
when you execute your select without function it will return all lines, but in function you stick the result of select in one variable. And if select in fuction will return to you two lines then it will throw exception, because Oracle doesn't manage to stick two lines for one variable.
So, you have two different ways: throw throw exception from you function like that:
create or replace function show_speed
(in_nplate varchar2)
return number
is
ret number(4);
begin
select speed into ret from cars where in_nplate = nplate;
return ret;
exception when too_many_rows than
return ret;
end show_speed;
Or another way: process exception with for .. loop
create or replace function show_speed
(in_nplate varchar2)
return number
is
ret varchar(4);
begin
for x in (select speed from cars where in_nplate = nplate)
loop
ret:= ret||','||x.speed;
end loop;
return ret;
end show_speed;
but this varians is not compatible with your expectings.

Postgresql function auto add 's' character to the end of string

I have a function in PostgreSQL database. However, every time I run the function in PostgreSQL, it auto-adda the character s behind of the query, therefore, I receive an error when it executes.
An example query ends up looking like this:
WHAT IN HERE: query SELECT uom_id FROM ps_quantity where id = 11s
My version is PostgreSQL 9.2.13. How can I solve this? Thanks.
CREATE OR REPLACE FUNCTION select_field(
selected_table text,
selected_field text,
field_type_sample anyelement,
where_clause text DEFAULT ''::text)
RETURNS anyelement AS
$BODY$ DECLARE
-- Log
ME constant text := 'selected_field()';
-- Local variables
_qry varchar := '';
_result_value ALIAS FOR $0;
where_clause1 varchar := 'asdasdsad';
BEGIN
RAISE NOTICE 'FUNCTION: SELECT_FIELD';
-- BANGPH
RAISE NOTICE 'BANGPH - CHANGE 11s to 11';
_qry := 'SELECT uom_id FROM ps_quantity where id = 11';
RAISE NOTICE 'WHERE = %s', where_clause1;
RAISE NOTICE 'WHAT IN HERE: query %s', _qry;
As the manual explains:
instead of:
RAISE NOTICE 'WHAT IN HERE: query %s', _qry;
you need to use:
RAISE NOTICE 'WHAT IN HERE: query %', _qry;
Placeholders for the RAISE statement don't have a "type modifier", it's a plain %

Adding Many (UDFs) Validation Functions to Oracle - Which Method Run Fastest

I have to move around 50+ validation functions into Oracle. I'm looking for the approach that runs fastest, but also would like to get around a boolean issue if possible. The return object for them all needs to be the same so that the application can react off the result in a consistent fashion and alert the user or display whatever popups, messages we may need. I created a valObj for this, but not sure yet if that is the best approach. The return format can be changed because the front-end that reacts off of it is not developed yet. In the end it will contain many different validation functions, from integer, number, phone, email, IPv4, IPv6, etc... This is what I have so far...
/***
This is the validation object.
It stores 1 for valid, 0 for not valid and some helper text that can be relayed back to the user.
***/
create or replace type valObj as object (
result number(1),
resultText varchar(32000)
);
/***
Coming from ColdFusion this seems clean to me but the function
will end up being a couple thousand lines long.
***/
create or replace function isValid(v in varchar2, format in varchar2)
return valObj
is
test number;
begin
if format = 'number' then
begin
test := to_number(v);
return valObj(1,null);
exception when VALUE_ERROR then return valObj(0,'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...');
end;
elsif format = 'integer' then
null; --TO DO
elsif format = 'email' then
null; --TO DO
elsif format = 'IPv4' then
null; --TO DO
elsif format = 'IPv6' then
null; --TO DO
end if;
--dozens of others to follow....
end;
/
/* Example Usage in SQL */
select isValid('blah','number') from dual; -- returns: (0, Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...)
select isValid('blah','number').result from dual; -- returns: 0
select isValid('blah','number').resulttext from dual; -- returns: Valid formats are: 12345, 12345.67, -12345, etc...
select isValid(1234567890.123,'number') from dual; -- returns: 1,{null}
select isValid(1234567890.123,'number').result from dual; -- returns: 1
select isValid(1234567890.123,'number').resulttext from dual; -- returns: {null}
/* Example Usage in PL/SQL */
declare
temp valObj;
begin
temp := isValid('blah','number');
if (temp.result = 0) then
dbms_output.put_line(temp.resulttext);
else
dbms_output.put_line('Valid');
end if;
end;
/
My questions are:
When using it in PL/SQL I would love to be able to do boolean checks instead like this: if (temp.result) then but I can't figure out a way, cause that won't work in SQL. Should I just add a 3rd boolean attribute to the valObj or is there another way I don't know of?
These validation functions could end up being called within large loops. Knowing that, is this the most efficient way to accomplish these validations?
I'd appreciate any help. Thanks!
UPDATE: I forgot about MEMBER FUNCTIONS. Thanks #Brian McGinity for reminding me. So I'd like to go with this method since it keeps the type and its functions encapsulated together. Would there be any speed difference between this method and a stand-alone function? Would this be compiled and stored the same as a stand-alone function?
create or replace type isValid as object (
result number(1),
resulttext varchar2(32000),
constructor function isValid(v varchar, format varchar) return self as result );
/
create or replace type body isValid as
constructor function isValid(v varchar, format varchar) return self as result as
test number;
begin
if format = 'number' then
begin
test := to_number(v);
self.result := 1;
self.resulttext := null;
return;
exception when VALUE_ERROR then
self.result := 0;
self.resulttext := 'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...';
return;
end;
elsif format = 'phone' then
null; --TO DO
end if;
--and many others...
end;
end;
/
/* Example Usage in SQL */
select isValid('a','number') from dual;
/* Example Usage in PL/SQL */
declare
begin
if (isValid('a','number').result = 1) then
null;
end if;
end;
/
TEST RESULTS:
/* Test isValid (the object member function), this took 7 seconds to run */
declare
begin
for i in 1 .. 2000000 loop
if (isValid('blah','number').result = 1) then
null;
end if;
end loop;
end;
/* Test isValid2 (the stand-alone function), this took 16 seconds to run */
declare
begin
for i in 1 .. 2000000 loop
if (isValid2('blah','number').result = 1) then
null;
end if;
end loop;
end;
Both isValid and isValid2 do the same exact code, they just run this line test := to_number(v); then do the exception if it fails and return the result. Does this appear to be a valid test? The Object member function method is actually faster than a stand-alone function???
The stand-alone function can be much faster if you set it to DETERMINISTIC and if the data is highly repetitive. On my machine this setting decreased run time from 9 seconds to 0.1 seconds. For reasons I don't understand that setting does not improve performance of the object function.
create or replace function isValid2(v in varchar2, format in varchar2)
return valObj
deterministic --<< Hit the turbo button!
is
test number;
begin
if format = 'number' then
begin
test := to_number(v);
return valObj(1,null);
exception when VALUE_ERROR then return valObj(0,'Invalid number. Valid formats are: 12345, 12345.67, -12345, etc...');
end;
end if;
end;
/
May also want to consider utilizing pls_integer over number. Don't know if it will buy you much, but documents suggest some gain will be had.
http://docs.oracle.com/cd/B10500_01/appdev.920/a96624/03_types.htm states,
"You use the PLS_INTEGER datatype to store signed integers. Its magnitude range is -2*31 .. 2*31. PLS_INTEGER values require less storage than NUMBER values. Also, PLS_INTEGER operations use machine arithmetic, so they are faster than NUMBER and BINARY_INTEGER operations, which use library arithmetic. For efficiency, use PLS_INTEGER for all calculations that fall within its magnitude range."

if-statement with string containing the condition

This question is about Postgresql 8.3.
I've got a table with a field containing conditions like 'lastcontact is null'. In code, I want to loop through this table and for each record, I want to check 'if condition then', like in this example:
FOR myrec IN
SELECT * FROM tabel ORDER BY colorlevel, volgnummer
LOOP
if (myrec.conditie) then
raise notice 'Condition % is true', myrec.conditie;
else
raise notice 'Condition % is false', myrec.conditie;
end if;
END LOOP;
The table which I have called 'tabel' in this example:
ID | Conditie | Colorlevel | Volgnummer | Code | Description
1 | lastcontact is null | 1 | 1 | ... | ...
2 | lastchanged is null | 1 | 2 | ... | ...
3 | lastmodified is null | 1 | 3 | ... | ...
Is it possible to do the check I desire? The code above results in the following error:
ERROR: invalid input syntax for type boolean: "lastcontact is null"
New section containing the result of Erwin's function
I have used this function:
CREATE OR REPLACE FUNCTION foo(lastcontact timestamptz)
RETURNS void AS
$BODY$
DECLARE
myrec record;
mycond boolean;
BEGIN
FOR myrec IN
SELECT * FROM tabel ORDER BY colorlevel, volgnummer
LOOP
EXECUTE 'SELECT ' || myrec.conditie || ' FROM tabel' INTO mycond;
IF mycond then
RAISE NOTICE 'Condition % is true', myrec.conditie;
ELSE
RAISE NOTICE 'Condition % is false', COALESCE(myrec.conditie, 'NULL');
END IF;
END LOOP;
END;
$BODY$
language 'plpgsql' volatile
cost 100;
I get this error:
ERROR: column "lastcontact" does not exist
LINE 1: SELECT lastcontact is null FROM tabel
^
QUERY: SELECT lastcontact is null FROM tabel
CONTEXT: PL/pgSQL function "foo" line 9 at EXECUTE statement1
I tried to find an explanation myself, but to no avail. It obviously is trying to run the statement against the database, but it should understand that 'lastcontact' is the variable that's been given as a function parameter.
From the comments I finally think I understand. You need dynamic SQL:
CREATE OR REPLACE FUNCTION foo(lastcontact timestamptz)
RETURNS void AS
$func$
DECLARE
myrec record;
mycond boolean;
BEGIN
FOR myrec IN
SELECT * FROM tabel ORDER BY colorlevel, volgnummer
LOOP
IF myrec.conditie ~~ '%lastcontact %' THEN -- special case for input param
myrec.conditie := replace (myrec.conditie
, 'lastcontact '
, CASE WHEN lastcontact IS NULL THEN 'NULL '
ELSE '''' || lastcontact::text || ''' ' END);
END IF;
EXECUTE 'SELECT ' || myrec.conditie || ' FROM tabel' INTO mycond;
IF mycond then
RAISE NOTICE 'Condition % is true', myrec.conditie;
ELSE
RAISE NOTICE 'Condition % is false', COALESCE(myrec.conditie, 'NULL');
END IF;
END LOOP;
END
$func$ LANGUAGE plpgsql;
Be aware, however, that this setup is wide open for SQL injection. Only use verified input.
A function works in PostgreSQL 8.3 as well (no DO statements, yet).
You cannot refer to parameters inside dynamic SQL (EXECUTE statement). You have to put the value into the query string.
In PostgreSQL 8.4 or later you have the superior commodity of the USING clause. Alas, not in version 8.3. You should consider upgrading if you can.
I put in a workaround for your old version. You have to take special care of the NULL value.