I am working on an extension of a PostgreSQL library that takes a string representation of a query as input. Basically I need to instantiate the resulting table that this string-based query produces, modify it, and then pass it to another function.
Right now I am just trying to get the query instatiated as a temporary table, so I am using this sample query:
CREATE TEMPORARY TABLE pgr_table (seq INTEGER, path_seq INTEGER, node INTEGER, edge BIGINT, cost DOUBLE PRECISION, agg_cost DOUBLE PRECISION);
EXECUTE 'SELECT gid AS id, source, target, cost, reverse_cost FROM ways;' INTO pgr_table;
But this results in a syntax error, just after the EXECUTE command. Am I not using it correctly?
By the way, I am aware of the dangers of SQL injection and using EXECUTE willy-nilly. The queries that I am making are not designed for front-end use, and I am following the design patterns already set forth by the library which I am modifying.
you confuse SQL execute and plpgsql execute - first to execute prepared statement and is run in SQL (as you try). second is a part of function plpgsql code
https://www.postgresql.org/docs/current/static/sql-execute.html
EXECUTE — execute a prepared statement
https://www.postgresql.org/docs/current/static/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN
Oftentimes you will want to generate dynamic commands inside your
PL/pgSQL functions, that is, commands that will involve different
tables or different data types each time they are executed. PL/pgSQL's
normal attempts to cache plans for commands (as discussed in Section
42.10.2) will not work in such scenarios. To handle this sort of problem, the EXECUTE statement is provided:
examples:
t=# prepare s as select now();
PREPARE
t=# execute s;
now
-------------------------------
2017-12-14 12:47:28.844485+00
(1 row)
and plpgsql:
t=# do
$$
declare
t text;
begin
execute 'select now()' into t;
raise info '%',t;
end;
$$
;
INFO: 2017-12-14 12:48:45.902768+00
DO
updtae
to avoid injection using dynamic code, use function format
https://www.postgresql.org/docs/current/static/functions-string.html
Format arguments according to a format string. This function is
similar to the C function sprintf.
Related
I have a procedure which takes as input a suffix of a table's name. Then, using execute format(), I pass this parameter to execute the dynamic query. The problem is that this parameter is the same throughout - I do not want to pass it x times as such:
execute format('SELECT table_%s.field1, table_%s.field2,table_%s.field3
FROM table_%s', inTableSuffix, inTableSuffix, inTableSuffix, inTableSuffix, ...)
I would like a format similar to the following:
execute format('SELECT table_%s.field1, table_%s.field2,table_%s.field3
FROM table_%s', inTableSuffix)
I understand that I can solve this problem using an alias to the table name but is there another alternative?
You can repeatedly address the positional arguments like this:
execute format('SELECT table_%1$s.field1
, table_%1$s.field2,table_%1$s.field3
FROM table_%1$s;', inTableSuffix);
Note: in this particular case, you can avoid repeating yourself by using an alias:
execute format('SELECT tx.field1
, tx.field2, tx.field3
FROM table_%s tx;', inTableSuffix);
#wildplasser already provided the syntax how to reference the same parameter repeatedly in format(). The code is still dangerous, though. You need to escape identifiers built from user input:
EXECUTE format('SELECT field1, field2, field3 FROM %I', 'table_' || inTableSuffix);
For the given example, you only need the parameter once anyway. The scope of dynamic queries executed with EXECUTE in plpgsql is limited to the query itself. Other variables or parameters of the function are not visible inside EXECUTE. Hence, there is not need to table-qualify columns in a dynamic query with a single table in the FROM clause.
But more importantly, %I is used in format() for identifiers to avoid syntax errors, misguided lower-casing or, worse, SQL injection! It double-quotes (only!) otherwise illegal identifiers, just like quote_ident() would.
Details:
SQL injection in Postgres functions vs prepared queries
Demonstrate SQL injection in PL/pgSQL
Are PostgreSQL column names case-sensitive?
I have an SQL statement in Oracle SQL developer that has some variables:
DEFINE custom_date = "'22-JUL-2016'" --run1
DEFINE custom_date = "'25-JUL-2016'" --run2
SELECT * FROM TABLE WHERE date=&custom_date
The real query is much more complicated and has many more variables and new tables are created from the results of the query. How can I create a script so that the query is executed twice, the first time with the custom date set as in the first line and the second time as in the second line.
In Oracle, the &variable is a "substitution variable" and is not part of SQL; it is part of the SQL*Plus scripting language (understood by SQL Developer, Toad etc.)
The better option, and what you are asking about, is BIND VARIABLES. The notation is :variable (with a colon : instead of &), for example :custom_date.
The difference is that a substitution variable is replaced by its value in the front-end application (SQL Developer in your case) before the query is ever sent to the Oracle engine proper. A bind variable is substituted at runtime. This has several benefits; discussing them is outside the scope of your question.
When you execute a query with bind variables in SQL Developer, the program will open a window where you enter the desired values for the bind variables. You will have to experiment with that a little bit till you can make it work (for example I never remember if a date must be entered with the single quotes or without). Good luck!
Define is used in TRANSACT SQL. To do this Oracle way, You can create anonymus PL/SQL block, similar to this:
DECLARE
p_param1 DATE;
p_param2 NUMBER;
CURSOR c_cur1(cp_param1 DATE,cp_param2 NUMBER)
IS
SELECT * FROM table WHERE date = cp_param1
;
BEGIN
-- Execute it first time
p_param1 := TO_DATE('2016-09-01','YYYY-MM-DD');
FOR r IN c_cur1(p_param1)
LOOP
NULL;
END LOOP;
-- Execute it second time
p_param1 := TO_DATE('2015-10-11','YYYY-MM-DD');
FOR r IN c_cur1(p_param1)
LOOP
NULL;
END LOOP;
END;
And in it, You create cursor with parameters and execute it twice with different parameter.
I do not know why You want to execute this query twice, so the script abowe does nothing with results, but it certainly should execute Your query twice, with different params.
Am very new in Database development so I have some doubts regarding my following example:
Function f1() - language sql
create or replace function f1(istr varchar)
returns text as $$
select 'hello! '::varchar || istr;
$$ language sql;
Function f2() - language plpgsql
create or replace function f2(istr varchar)
returns text as $$
begin select 'hello! '::varchar || istr; end;
$$ language plpgsql;
Both functions can be called like select f1('world') or select f2('world').
If I call select f1('world') the output will be:
`hello! world`
And output for select f2('world'):
ERROR: query has no destination for result data
HINT: If you want to discard the results of a SELECT, use PERFORM instead.
CONTEXT: PL/pgSQL function f11(character varying) line 2 at SQL statement
********** Error **********
I wish to know the difference and in which situations I should use language sql or language plpgsql.
Any useful link or answers regarding functions will much appreciated.
SQL functions
... are the better choice:
For simple scalar queries. Not much to plan, better save any overhead.
For single (or very few) calls per session. Nothing to gain from plan caching via prepared statements that PL/pgSQL has to offer. See below.
If they are typically called in the context of bigger queries and are simple enough to be inlined.
For lack of experience with any procedural language like PL/pgSQL. Many know SQL well and that's about all you need for SQL functions. Few can say the same about PL/pgSQL. (Though it's rather simple.)
A bit shorter code. No block overhead.
PL/pgSQL functions
... are the better choice:
When you need any procedural elements or variables that are not available in SQL functions, obviously.
For any kind of dynamic SQL, where you build and EXECUTE statements dynamically. Special care is needed to avoid SQL injection. More details:
Postgres functions vs prepared queries
When you have computations that can be reused in several places and a CTE can't be stretched for the purpose. In an SQL function you don't have variables and would be forced to compute repeatedly or write to a table. This related answer on dba.SE has side-by-side code examples for solving the same problem using an SQL function / a plpgsql function / a query with CTEs:
How to pass a parameter into a function
Assignments are somewhat more expensive than in other procedural languages. Adapt a programming style that doesn't use more assignments than necessary.
When a function cannot be inlined and is called repeatedly. Unlike with SQL functions, query plans can be cached for all SQL statements inside a PL/pgSQL functions; they are treated like prepared statements, the plan is cached for repeated calls within the same session (if Postgres expects the cached (generic) plan to perform better than re-planning every time. That's the reason why PL/pgSQL functions are typically faster after the first couple of calls in such cases.
Here is a thread on pgsql-performance discussing some of these items:
Re: pl/pgsql functions outperforming sql ones?
When you need to trap errors.
For trigger functions.
When including DDL statements changing objects or altering system catalogs in any way relevant to subsequent commands - because all statements in SQL functions are parsed at once while PL/pgSQL functions plan and execute each statement sequentially (like a prepared statement). See:
Why can PL/pgSQL functions have side effect, while SQL functions can't?
Also consider:
PostgreSQL Stored Procedure Performance
To actually return from a PL/pgSQL function, you could write:
CREATE FUNCTION f2(istr varchar)
RETURNS text AS
$func$
BEGIN
RETURN 'hello! '; -- defaults to type text anyway
END
$func$ LANGUAGE plpgsql;
There are other ways:
Can I make a plpgsql function return an integer without using a variable?
The manual on "Returning From a Function"
PL/PgSQL is a PostgreSQL-specific procedural language based on SQL. It has loops, variables, error/exception handling, etc. Not all SQL is valid PL/PgSQL - as you discovered, for example, you can't use SELECT without INTO or RETURN QUERY. PL/PgSQL may also be used in DO blocks for one-shot procedures.
sql functions can only use pure SQL, but they're often more efficient and they're simpler to write because you don't need a BEGIN ... END; block, etc. SQL functions may be inlined, which is not true for PL/PgSQL.
People often use PL/PgSQL where plain SQL would be sufficient, because they're used to thinking procedurally. In most cases when you think you need PL/PgSQL you probably actually don't. Recursive CTEs, lateral queries, etc generally meet most needs.
For more info ... see the manual.
just make the select query you wrote inside the function as the returned value:
create or replace function f2(istr varchar)
returns text as $$
begin return(select 'hello! '::varchar || istr); end;
$$ language plpgsql;
I am trying to convert a big list of SQL statements into a PostgreSQL stored procedure that uses a variable, one that should be populated from the result of one SELECT.
If you want to see what has to be run, you can check it here
As far as I know PostgreSQL does not allow use to use variables inside stored procedures that are using SQL language, so I'm looking for solutions that would require a minimal number of changes.
It's much easier after you find the right syntax:
Here is the procedure definition for plpgsql language:
DECLARE myvar integer;
BEGIN
SELECT INTO myvar FROM ...;
-- use myvar
END;
The code seems to be pretty repetitive. Will EXECUTE be of any help? (manual about execute) (example and more information) It allows you to run predefined queries and create new ones on the fly.
I write PSQL script and using variables (for psql --variable key=value commandline syntax).
This works perfectly for top level scope like select * from :key, but I create functions with the script and need variable value inside them.
So, the syntax like
create function foo() returns void as
$$
declare
begin
grant select on my_table to group :user;
end;
$$
language plpgsql;
fails at :user.
As far as I understand psql variables is a plain macro substitution feature, but it doesn't process function bodies.
Are there any escaping syntax for such cases? Surrounding :user with $$ works regarding substitution, but psql fails at $$.
Are there any other way to do this besides standalone macro processing (sed, awk, etc)?
PSQL SET variables aren't interpolated inside dollar-quoted strings. I don't know this for certain, but I think there's no escape or other trickery to turn on SET variable interpolation in there.
One might think you could wedge an unquoted :user between two dollar-quoted stretches of PL/pgSQL to get the desired effect. But this doesn't seem to work... I think the syntax requires a single string and not an expression concatenating strings. Might be mistaken on that.
Anyway, that doesn't matter. There's another approach (as Pasco noted): write the stored procedure to accept a PL/pgSQL argument. Here's what that would look like.
CREATE OR REPLACE FUNCTION foo("user" TEXT) RETURNS void AS
$$
BEGIN
EXECUTE 'GRANT SELECT ON my_table TO GROUP ' || quote_ident(user);
END;
$$ LANGUAGE plpgsql;
Notes on this function:
EXECUTE generates an appropriate GRANT on each invocation using on our procedure argument. The PG manual section called "Executing Dynamic Commands" explains EXECUTE in detail.
The declaration of procedure argument user must be double quoted. Double quotes force it to be interpreted as an identifier.
Once you define the function like this, you can call it using interpolated PSQL variables. Here's an outline.
Run psql --variable user="'whoever'" --file=myscript.sql. Single quotes are required around the username!
In myscript.sql, define function like above.
In myscript.sql, put select foo(:user);. This is where we rely on those single quotes we put in the value of user.
Although this seems to work, it strikes me as rather squirrely. I thought SET variables were intended for runtime configuration. Carrying data around in SET seems odd.
Edit: here's a concrete reason to not use SET variables. From the manpage: "These assignments are done during a very early stage of startup, so variables reserved for internal purposes might get overwritten later." If Postgres decided to use a variable named user (or whatever you pick), it could overwrite your script argument with something you never intended. In fact, psql already takes USER for itself -- this only works because SET is case sensitive. This very nearly broke things from the start!
You could use the method Dan LaRocque describes to make it kind of hack'ishly work (not a knock at Dan) - but psql is not really your friend for doing this kind of work (assuming this kind of work is scripting and not one-off things). If you've got a function, rewrite it to take a parameter like so:
create function foo(v_user text) returns void as
$$
begin
execute 'grant select on my_table to group '||quote_ident(v_user);
end;
$$
language plpgsql;
(quote_ident() makes it so you don't have to assure the double-quotes, it handles all that.)
But then use a real scripting language like Perl, Python, Ruby, etc that has a Postgres driver to invoke your function with a parameter.
Psql has its place, I'm just not sure that this is it.
Actually, there's a way in more modern versions of psql.
Since the function body has to be a string constant and psql does not perform variable interpolation within string constants we have to resort to assembling the function body in a separate first step:
select format($gexec$
create function foo() returns void as
$$
begin
grant select on my_table to group %I;
end;
$$
language plpgsql;
$gexec$, :'user') \gexec
The trailing \gexec command
executes the "outer" query using the format function to produce the desired query sting and then
executes the produced query string as a query.