I have a rather large view constructed using the "WITH" statement. I've build the view's logic using small, understandable sub queries which are built on top of each other.
The result is a clearly structured SQL which (I believe ) is good to follow even if you're not the creator.
My problem comes with debugging in the future. If at some stage a collegue wants to understand how the result of the view is computed, it's a good approach to to execute some of the sub queries.
The usual approach would be to copy the view's SQL to an SQL-editor (e.g. SQL-Developer) and to replace the main statement with the subquery you're interested in.
Example:
WITH
all_orders AS (
SELECT order, price ... FROM ...
),
all_customers AS (
SELECT id,
last_name,
first_name,
first_order_date ...
FROM...
),
new_customers AS (
SELECT id,
last_name,
first_name
FROM all_customers
WHERE first_order_date > ...
)
-- main SQL
SELECT ... FROM all_orders a
INNER JOIN new_customers ON (...)
If I have the feeling that something is wrong with "new_customers" I would comment out the main SQL and replace it with:
...
-- main SQL
-- SELECT ... FROM all_orders a
-- INNER JOIN new_customers ON (...)
SELECT * FROM new_customers;
If I see that new_customers contains wrong data and I want to check if at least it's source "all_customers" is correct, I replace my main SQL again with
...
-- main SQL
-- SELECT ... FROM all_orders a
-- INNER JOIN new_customers ON (...)
SELECT * FROM all_customers;
That works really well but as soon as the SQL is inside a view I only have access to the result of the main SQL as the normal output of the view.
However, for simple debugging (meaning without going to the SQL editor, looking up the views definition and coping the SQL to the SQL editor) it would be really helpful to have some kind of database function where I could say:
SELECT * FROM RUN_SUBQUERY('my_view_name', 'new_customers');
My question: Is there such a database function or some similar approach which would allow me to quickly execute sub queries of a database view without splitting the logic up into sub views?
See below for my experience with sub views.
Alternatives
Split the big view in separate smaller views:
I tried this. The execution speed of the SQL dropped by factor 10. Since this is way too slow I'm also looking at some possible optimization here - however, I've seen it running really well when it was all in one statement/ view so it's hard to justify the extra effort here. Again, I only need the sub query results for debugging.
Keep the big view and additionally split it into smaller views which are just used for debugging:
This might be a way to go but we all know that having logic defined in two places is never a good idea (DRY).
You could use a function that returns a table of records.
In this function you can according to the input parameters construct your sql differently and return the result with a cursor.
So for example:
select * from table(cast(run_subquery('my_view_name','new_customers' as a_table_of_records_type));
and this calls the function run_subquery:
function RUN_SUBQUERY (VNAME IN VARCHAR2,QNAME IN VARCHAR2)
return a_table_of_records_type is
query_string varchar2(4000);
begin
if VNAME = '...' then
query_string := query_string || '....';
end if;
if VNAME = '...' then
query_string := query_string || '....';
end if;
-- Execute string through refcursor and put the output in a_table_of_records_type
return a_table_of_records_type;
end;
Related
I have a table similar to this:
CREATE TABLE example (
id integer primary key,
name char(200),
parentid integer,
value integer);
I can use the parentid field to arrange data into a tree structure.
Now here's the bit I can't work out. Given a parentid, is it possible to write an SQL statement to add up all the value fields under that parentid and recurse down the branch of the tree ?
UPDATE: I'm using posgreSQL so the fancy MS-SQL features are not available to me. In any case, I'd like this to be treated as a generic SQL question.
BTW, I'm very impressed to have 6 answers within 15 minutes of asking the question! Go stack overflow!
Here is an example script using common table expression:
with recursive sumthis(id, val) as (
select id, value
from example
where id = :selectedid
union all
select C.id, C.value
from sumthis P
inner join example C on P.id = C.parentid
)
select sum(val) from sumthis
The script above creates a 'virtual' table called sumthis that has columns id and val. It is defined as the result of two selects merged with union all.
First select gets the root (where id = :selectedid).
Second select follows the children of the previous results iteratively until there is nothing to return.
The end result can then be processed like a normal table. In this case the val column is summed.
Since version 8.4, PostgreSQL has recursive query support for common table expressions using the SQL standard WITH syntax.
If you want a portable solution that will work on any ANSI SQL-92 RDBMS, you will need to add a new column to your table.
Joe Celko is the original author of the Nested Sets approach to storing hierarchies in SQL. You can Google "nested sets" hierarchy to understand more about the background.
Or you can just rename parentid to leftid and add a rightid.
Here is my attempt to summarize Nested Sets, which will fall woefully short because I'm no Joe Celko: SQL is a set-based language, and the adjacency model (storing parent ID) is NOT a set-based representation of a hierarchy. Therefore there is no pure set-based method to query an adjacency schema.
However, most of the major platforms have introduced extensions in recent years to deal with this precise problem. So if someone replies with a Postgres-specific solution, use that by all means.
There are a few ways to do what you need in PostgreSQL.
If you can install modules, look at the tablefunc contrib. It has a connectby() function that handles traversing trees. http://www.postgresql.org/docs/8.3/interactive/tablefunc.html
Also check out the ltree contrib, which you could adapt your table to use: http://www.postgresql.org/docs/8.3/interactive/ltree.html
Or you can traverse the tree yourself with a PL/PGSQL function.
Something like this:
create or replace function example_subtree (integer)
returns setof example as
'declare results record;
child record;
begin
select into results * from example where parent_id = $1;
if found then
return next results;
for child in select id from example
where parent_id = $1
loop
for temp in select * from example_subtree(child.id)
loop
return next temp;
end loop;
end loop;
end if;
return null;
end;' language 'plpgsql';
select sum(value) as value_sum
from example_subtree(1234);
A standard way to make a recursive query in SQL are recursive CTE. PostgreSQL supports them since 8.4.
In earlier versions, you can write a recursive set-returning function:
CREATE FUNCTION fn_hierarchy (parent INT)
RETURNS SETOF example
AS
$$
SELECT example
FROM example
WHERE id = $1
UNION ALL
SELECT fn_hierarchy(id)
FROM example
WHERE parentid = $1
$$
LANGUAGE 'sql';
SELECT *
FROM fn_hierarchy(1)
See this article:
Hierarchical queries in PostgreSQL
If your using SQL Server 2005, there is a really cool way to do this using Common Table Expressions.
It takes all of the gruntwork out of creating a temporary table, and basicly allows you to do it all with just a WITH and a UNION.
Here is a good tutorial:
http://searchwindevelopment.techtarget.com/tip/0,289483,sid8_gci1278207,00.html
use a common table expression.
May want to indicate this is SQL Server 2005 or above only. Dale Ragan
here's an article on recursion by SqlTeam without common table expressions.
The following code compiles and it's tested OK.
create or replace function subtree (bigint)
returns setof example as $$
declare
results record;
entry record;
recs record;
begin
select into results * from example where parent = $1;
if found then
for entry in select child from example where parent = $1 and child parent loop
for recs in select * from subtree(entry.child) loop
return next recs;
end loop;
end loop;
end if;
return next results;
end;
$$ language 'plpgsql';
The condition "child <> parent" is needed in my case because nodes point to themselves.
Have fun :)
Oracle has "START WITH" and "CONNECT BY"
select
lpad(' ',2*(level-1)) || to_char(child) s
from
test_connect_by
start with parent is null
connect by prior child = parent;
http://www.adp-gmbh.ch/ora/sql/connect_by.html
Just as a brief aside although the question has been answered very well, it should be noted that if we treat this as a:
generic SQL question
then the SQL implementation is fairly straight-forward, as SQL'99 allows linear recursion in the specification (although I believe no RDBMSs implement the standard fully) through the WITH RECURSIVE statement. So from a theoretical perspective we can do this right now.
None of the examples worked OK for me so I've fixed it like this:
declare
results record;
entry record;
recs record;
begin
for results in select * from project where pid = $1 loop
return next results;
for recs in select * from project_subtree(results.id) loop
return next recs;
end loop;
end loop;
return;
end;
is this SQL Server? Couldn't you write a TSQL stored procedure that loops through and unions the results together?
I am also interested if there is a SQL-only way of doing this though. From the bits I remember from my geographic databases class, there should be.
I think it is easier in SQL 2008 with HierarchyID
If you need to store arbitrary graphs, not just hierarchies, you could push Postgres to the side and try a graph database such as AllegroGraph:
Everything in the graph database is stored as a triple (source node, edge, target node) and it gives you first class support for manipulating the graph structure and querying it using a SQL like language.
It doesn't integrate well with something like Hibernate or Django ORM but if you are serious about graph structures (not just hierarchies like the Nested Set model gives you) check it out.
I also believe Oracle has finally added a support for real Graphs in their latest products, but I'm amazed it's taken so long, lots of problems could benefit from this model.
If I have two queries, which I will call horrible_query_1 and ugly_query_2, and I want to perform the following two minus operations on them:
(horrible_query_1) minus (ugly_query_2)
(ugly_query_2) minus (horrible_query_1)
Or maybe I have a terribly_large_and_useful_query, and the result set it produces I want to use as part of several future queries.
How can I avoid copying and pasting the same queries in multiple places? How can I "not repeat myself," and follow DRY principles. Is this possible in SQL?
I'm using Oracle SQL. Portable SQL solutions are preferable, but if I have to use an Oracle specific feature (including PL/SQL) that's OK.
create view horrible_query_1_VIEW as
select .. ...
from .. .. ..
create view ugly_query_2_VIEW as
select .. ...
from .. .. ..
Then
(horrible_query_1_VIEW) minus (ugly_query_2_VIEW)
(ugly_query_2_VIEW) minus (horrible_query_1_VIEW)
Or, maybe, with a with clause:
with horrible_query_1 as (
select .. .. ..
from .. .. ..
) ,
ugly_query_2 as (
select .. .. ..
.. .. ..
)
(select * from horrible_query_1 minus select * from ugly_query_2 ) union all
(select * from ugly_query_2 minus select * from horrible_query_1)
If you want to reuse the SQL text of the queries, then defining views is the best way, as described earlier.
If you want to reuse the result of the queries, then you should consider global temporary tables. These temporary tables store data for the duration of session or transaction (whichever you choose). These are really useful in case you need to reuse calculated data many times over, especially if your queries are indeed "ugly" and "horrible" (meaning long running). See Temporary tables for more information.
If you need to keep the data longer than a session, you can consider materialized views.
Since you're using Oracle, I'd create Pipelined TABLE functions.
The function takes parameters and returns an object (which you have to create)
and then you SELECT * or even specific columns from it using the TABLE() function and can use it with a WHERE clause or with JOINs. If you want a unit of reuse (a function) you're not restricted to just returning values (i.e a scalar function) you can write a function that returns rows or recordsets.
something like this:
FUNCTION RETURN_MY_ROWS(Param1 IN type...ParamX IN Type)
RETURN PARENT_OBJECT PIPELINED
IS
local_curs cursor_alias; --you need a cursor alias if this function is in a Package
out_rec ROW_RECORD_OF_CUSTOM_OBJECT:=ROW_RECORD_OF_CUSTOM_OBJECT(NULL, NULL,NULL) --one NULL for each field in the record sub-object
BEGIN
OPEN local_curs FOR
--the SELECT query that you're trying to encapsulate goes here
-- and it can be very detailed/complex and even have WITH () etc..
SELECT * FROM baseTable WHERE col1 = x;
-- now that you have captured the SELECT into a Cursor
-- here you put a LOOP to take what's in the cursor and put it in the
-- child object (that holds the individual records)
LOOP
FETCH local_curs --opening the ref-cursor
INTO out_rec.COL1,
out_rec.COL2,
out_rec.COL3;
EXIT WHEN local_curs%NOTFOUND;
PIPE ROW(out_rec); --piping out the Object
END LOOP;
CLOSE local_curs; -- always do this
RETURN; -- we're now done
END RETURN_MY_ROWS;
after you've done that, you can use it like so
SELECT * FROM TABLE(RETURN_MY_ROWS(val1, val2));
you can INSERT SELECT or even CREATE TABLE out of it , you can have it in joins.
two more things to mention:
--ROW_RECORD_OF_CUSTOM_OBJECT is something along these lines
CREATE or REPLACE TYPE ROW_RECORD_OF_CUSTOM_OBJECT AS OBJECT
(
col1 type;
col2 type;
...
colx type;
);
and PARENT_OBJECT is a table of the other object (with the field definitions) we just made
create or replace TYPE PARENT_OBJECT IS TABLE OF ROW_RECORD_OF_CUSTOM_OBJECT;
so this function needs two OBJECTs to support it, but one is a record, the other is a table of that record (you have to create the record first).
In a nutshell, the function is easy to write, you need a child object (with fields), and a parent object that will house that child object that is of type TABLE of the child object, and you open the original base-table fetching SQL into a SYS_REFCURSOR (which you may need to alias) if you're in a package and you read from that cursor from a loop into the individual records.
The function returns a type of PARENT_OBJECT but inside it packs the records sub-object with values from the cursor.
I hope this works for you (there may be permissioning issues with your DBA if you want to create OBJECTs and Table functions)*/
If you operate with values, you could write functions.
Here you find infos on how to do it. It basically works like writing a function in any language. You can define parameters and return values.
Which gives you the cool possibility to write code just once. Here is how you do it:
http://docs.oracle.com/cd/B19306_01/server.102/b14200/statements_5009.htm
Have you tried using RESULT_CACHE hint in your queries? Also, you could
ALTER SESSION SET RESULT_CACHE_MODE=FORCE
and see if it helps.
So we have some developers who went a little view happy. So now we have views that reference views that reference views, ad nauseum.
So what I want, in order to assist me in Tuning, is to expand these views.
I want a function that takes a string and returns a string. The input string is the query, the output string is the same query without views.
CREATE OR REPLACE VIEW myView AS
SELECT * FROM emp
Using function/stored procedure "F":
F('SELECT * FROM myView')
...would return:
SELECT * FROM ( SELECT * FROM emp)
Is there an Oracle package for this?
Does someone have code in:
either SQL or PL/SQL
In something else
Short answer:
Not at this time
Not that I'm aware of
UPDATE
It looks like Oracle 12c has exactly what you need: DBMS_UTILITY.expand_sql_text
http://docs.oracle.com/cd/E16655_01/appdev.121/e17602/d_util.htm#ARPLS73973
One problem with what you are proposing is that there are usually multiple ways that a query involving views can be rewritten, so simply expanding the text of the views won't necessarily tell you a lot about how the query is being executed.
Since your purpose is tuning, I would suggest that the execution plans for the queries will probably give you the information you really need. This won't show the rewritten query, but it will show you all the actual tables and how they are referenced in executing the query.
The best way I know of to view the actual execution plan is:
SELECT /*+ gather_plan_statistics */ * FROM myView
select * from table(dbms_xplan.display_cursor(null,null,'ALLSTATS LAST'))
There are some methods to expand views but the output is so ugly it may not be useful.
12c Has a new procedure named DBMS_UTILITY.EXPAND_SQL_TEXT.
11g Has an undocumented procedure dbms_sql2.expand_sql_text.
Below is a simple example. The output may be helpful for the optimizer but it is probably not much use if you need a human-readable SQL statement.
create or replace view view1 as select 1 a, 2 b from dual;
create or replace view view2 as select a from view1;
declare
v_output clob;
begin
dbms_utility.expand_sql_text('select * from view2', v_output);
dbms_output.put_line(v_output);
end;
/
Output:
SELECT "A1"."A" "A" FROM (SELECT "A2"."A" "A" FROM (SELECT 1 "A",2 "B" FROM "SYS"."DUAL" "A3") "A2") "A1"
I have quite complicated view in mysql, like
select filter.id as filter_id, person.id, person.name
from person, filter
inner join
...
left join
...
where person_match_filter_condition ...
group by filter.filter_id, person.id, person.name
Query filters person which corresponds domain specific conditions.
Typical use of view is:
select * from where filter_id = some_value
Problem is that mysql cannot optimize query. It applies confition by filter_id AFTER get data for all filters - very ineffective.
Idea to get filter_id from other tables is not good for my case.
How can I transform my query to make it more effective?
Wrap the long query in a procedure, and pass the filters to the procedure call as parameters. Then instead of using views you call the procedure, the procedure will build you the entire query and will run optimized query.
Better yet, you can pass parameters to your views in a simple manner by creating a Function to GET your values from Session Variables. See https://www.stackoverflow.com/questions/14511760 for the technique. This is a copy of my create function you may wish to pattern after.
DELIMITER //
CREATE FUNCTION fn_getcase_id()
RETURNS MEDIUMINT(11)
DETERMINISTIC NO SQL
BEGIN
see stackoverflow.com/questions/14511760 and read ALL the info TWICE or MORE. wh 04/13/2017
RETURN #sv_case_id;
END//
DELIMITER ;
You will need to create a similar FN (one for each variable).
I have a table similar to this:
CREATE TABLE example (
id integer primary key,
name char(200),
parentid integer,
value integer);
I can use the parentid field to arrange data into a tree structure.
Now here's the bit I can't work out. Given a parentid, is it possible to write an SQL statement to add up all the value fields under that parentid and recurse down the branch of the tree ?
UPDATE: I'm using posgreSQL so the fancy MS-SQL features are not available to me. In any case, I'd like this to be treated as a generic SQL question.
BTW, I'm very impressed to have 6 answers within 15 minutes of asking the question! Go stack overflow!
Here is an example script using common table expression:
with recursive sumthis(id, val) as (
select id, value
from example
where id = :selectedid
union all
select C.id, C.value
from sumthis P
inner join example C on P.id = C.parentid
)
select sum(val) from sumthis
The script above creates a 'virtual' table called sumthis that has columns id and val. It is defined as the result of two selects merged with union all.
First select gets the root (where id = :selectedid).
Second select follows the children of the previous results iteratively until there is nothing to return.
The end result can then be processed like a normal table. In this case the val column is summed.
Since version 8.4, PostgreSQL has recursive query support for common table expressions using the SQL standard WITH syntax.
If you want a portable solution that will work on any ANSI SQL-92 RDBMS, you will need to add a new column to your table.
Joe Celko is the original author of the Nested Sets approach to storing hierarchies in SQL. You can Google "nested sets" hierarchy to understand more about the background.
Or you can just rename parentid to leftid and add a rightid.
Here is my attempt to summarize Nested Sets, which will fall woefully short because I'm no Joe Celko: SQL is a set-based language, and the adjacency model (storing parent ID) is NOT a set-based representation of a hierarchy. Therefore there is no pure set-based method to query an adjacency schema.
However, most of the major platforms have introduced extensions in recent years to deal with this precise problem. So if someone replies with a Postgres-specific solution, use that by all means.
There are a few ways to do what you need in PostgreSQL.
If you can install modules, look at the tablefunc contrib. It has a connectby() function that handles traversing trees. http://www.postgresql.org/docs/8.3/interactive/tablefunc.html
Also check out the ltree contrib, which you could adapt your table to use: http://www.postgresql.org/docs/8.3/interactive/ltree.html
Or you can traverse the tree yourself with a PL/PGSQL function.
Something like this:
create or replace function example_subtree (integer)
returns setof example as
'declare results record;
child record;
begin
select into results * from example where parent_id = $1;
if found then
return next results;
for child in select id from example
where parent_id = $1
loop
for temp in select * from example_subtree(child.id)
loop
return next temp;
end loop;
end loop;
end if;
return null;
end;' language 'plpgsql';
select sum(value) as value_sum
from example_subtree(1234);
A standard way to make a recursive query in SQL are recursive CTE. PostgreSQL supports them since 8.4.
In earlier versions, you can write a recursive set-returning function:
CREATE FUNCTION fn_hierarchy (parent INT)
RETURNS SETOF example
AS
$$
SELECT example
FROM example
WHERE id = $1
UNION ALL
SELECT fn_hierarchy(id)
FROM example
WHERE parentid = $1
$$
LANGUAGE 'sql';
SELECT *
FROM fn_hierarchy(1)
See this article:
Hierarchical queries in PostgreSQL
If your using SQL Server 2005, there is a really cool way to do this using Common Table Expressions.
It takes all of the gruntwork out of creating a temporary table, and basicly allows you to do it all with just a WITH and a UNION.
Here is a good tutorial:
http://searchwindevelopment.techtarget.com/tip/0,289483,sid8_gci1278207,00.html
use a common table expression.
May want to indicate this is SQL Server 2005 or above only. Dale Ragan
here's an article on recursion by SqlTeam without common table expressions.
The following code compiles and it's tested OK.
create or replace function subtree (bigint)
returns setof example as $$
declare
results record;
entry record;
recs record;
begin
select into results * from example where parent = $1;
if found then
for entry in select child from example where parent = $1 and child parent loop
for recs in select * from subtree(entry.child) loop
return next recs;
end loop;
end loop;
end if;
return next results;
end;
$$ language 'plpgsql';
The condition "child <> parent" is needed in my case because nodes point to themselves.
Have fun :)
Oracle has "START WITH" and "CONNECT BY"
select
lpad(' ',2*(level-1)) || to_char(child) s
from
test_connect_by
start with parent is null
connect by prior child = parent;
http://www.adp-gmbh.ch/ora/sql/connect_by.html
Just as a brief aside although the question has been answered very well, it should be noted that if we treat this as a:
generic SQL question
then the SQL implementation is fairly straight-forward, as SQL'99 allows linear recursion in the specification (although I believe no RDBMSs implement the standard fully) through the WITH RECURSIVE statement. So from a theoretical perspective we can do this right now.
None of the examples worked OK for me so I've fixed it like this:
declare
results record;
entry record;
recs record;
begin
for results in select * from project where pid = $1 loop
return next results;
for recs in select * from project_subtree(results.id) loop
return next recs;
end loop;
end loop;
return;
end;
is this SQL Server? Couldn't you write a TSQL stored procedure that loops through and unions the results together?
I am also interested if there is a SQL-only way of doing this though. From the bits I remember from my geographic databases class, there should be.
I think it is easier in SQL 2008 with HierarchyID
If you need to store arbitrary graphs, not just hierarchies, you could push Postgres to the side and try a graph database such as AllegroGraph:
Everything in the graph database is stored as a triple (source node, edge, target node) and it gives you first class support for manipulating the graph structure and querying it using a SQL like language.
It doesn't integrate well with something like Hibernate or Django ORM but if you are serious about graph structures (not just hierarchies like the Nested Set model gives you) check it out.
I also believe Oracle has finally added a support for real Graphs in their latest products, but I'm amazed it's taken so long, lots of problems could benefit from this model.