I have a particularly complicated query for a report. It selects several columns from a view, and it must build a column via aggregately concatenating several fields. To complicate things further, the concatenation must contain 3 fields even if there are 0 in reality (The concatenation is comma delimited, so empty fields will still be noticed).
We are using Oracle 11.1.0.7.0.
To provide backward compatibility (not necessary) we used the xmlagg function to perform the concatenation, I believe that has been around since Oracle 8 or 9.
This example will be simplified but I feel provides sufficient information. In other words, please do not focus on normalizing the table structure, this is strictly an example.
person_view
-----------
name
phone
address
position_id
position_table
--------------
position_id
position_title
So the query we currently have, and I admit to not being a SQL guru, is something like:
select
name,
phone,
address,
(select
xmlagg(xmlelement(e, position_title || ',')).extract('//text()')
from
(select
position_title
from
position_table
where
position_table.position_id = person_view.position_id and
rownum <= 3
union all select '' from dual
union all select '' from dual
union all select '' from dual
)
where
rownum <= 3
)
from
person_view
My actual error is that, it seems, the subquery that ensures at least 3 rows of input cannot reference the grandparents query to determine person_view.position_id.
I get ORA-00904: "PERSON_VIEW"."POSITION_ID": invalid identifier
Performance is not a huge concern, as this is a report that will not be run regularly, but I need to figure out a solution to aggregate this data with an absolute 3 columns of data. Any guidance to help rewrite the query, or allow the subquery to access the relevant grandparent column is greatly appreciated.
This is a limitation in Oracle SQL: you can't reference a parent query element from a subquery more than 1 level deep.
I would use a function in such a case:
CREATE OR REPLACE FUNCTION get_title(p_position_id NUMBER) RETURN VARCHAR2 IS
l_result LONG;
l_position_num NUMBER := 0;
BEGIN
FOR cc IN (SELECT position_title
FROM position_table
WHERE position_table.position_id = p_position_id
AND rownum <= 3) LOOP
l_result := cc.position_title || ',';
l_position_num := l_position_num + 1;
END LOOP;
RETURN l_result || rpad(',', 3 - l_position_num, ',');
END;
You query would look like this:
select
name,
phone,
address,
get_title(p.position_id) title
from person_view p
Related
The following code creates a filter that I successfully append to an existing sql statement.
DECLARE V_FILTER VARCHAR2(255);
begin
SELECT DISTINCT 'reserve reporting_unit_code' ||' IN (' ||
listagg ( chr(39) || uic || chr(39) , ',') within group (order by uic) || ')'
into v_FILTER
from ( select distinct uicf.uic
FROM OWSADHOC.ADHOC_UIC_FILTER UICF
INNER JOIN OWSADHOC.ADHOC_USER_ROLE UROLE
ON UICF.ROLE_NAME = UROLE.ROLE_NAME
inner join OWSADHOC.ADHOC_USER_DATA uname
on uname.USER_ID = UROLE.USER_ID
WHERE UROLE.ROLE_NAME in (:P_LoggedInUserRoles)
and uname.JASPER_ACCOUNT = :P_LoggedInUserName
)
;
DBMS_OUTPUT.PUT_LINE ('GENERATED FILTER: ' || V_FILTER);
end;
v_FILTER is reserve reporting_unit_code IN ('86749')
This works fine when P_LoggedInUserRoles has only one value. However, if I have a comma separated list in the variable, it is not working. I've tried defining expressionList as an array. PL/SQL doesn't compile using that method.
Is the expressionList constrained to the use of literal values if one has more than one value? I can't find that explicit statement in any manual, but I haven't been able to define the expressionList as a variable that is a collection of values.
No, you cannot, but you're quite close
A comma separated list will make the condition look like this:
WHERE UROLE.ROLE_NAME in ('Jane, Andrew, Jill')
and return an empty resultset. While what you're looking for is
WHERE UROLE.ROLE_NAME in ('Jane', 'Andrew', 'Jill')
It's quite clear that the number of bind variables will have to differ from one call to another as the lists have different length. Sorry, that's not something we can do easily. You'd need something called Dynamic SQL method 4 and that seems a bit of an overkill.
There's an easier way though. We're gonna parse the input string of comma separated values into a resultset. We're gonna do that in a subquery and we can use subqueries in IN.
Here's how we do it
WHERE UROLE.ROLE_NAME in (
SELECT NAME
FROM (
SELECT regexp_substr(:P_LoggedInUserRoles, '(.*?)(,|$)', 1, LEVEL, null, 1) NAME, LEVEL LVL
FROM DUAL
CONNECT BY LEVEL <= regexp_count(:P_LoggedInUserRoles, ',') + 1)
ORDER BY LVL)
I shamelessly stole the regexp from this question: Oracle- Split string comma delimited (string contains spaces and consecutive commas) so you might want to go ahead and upvote the answer there.
Now the downside is of course that we'll be doing this CONNECT BY trick all the time even if we received a single name, but that's a topic for another time.
I have (and don't own, so I can't change) a table with a layout similar to this.
ID | CATEGORIES
---------------
1 | c1
2 | c2,c3
3 | c3,c2
4 | c3
5 | c4,c8,c5,c100
I need to return the rows that contain a specific category id. I starting by writing the queries with LIKE statements, because the values can be anywhere in the string
SELECT id FROM table WHERE categories LIKE '%c2%';
Would return rows 2 and 3
SELECT id FROM table WHERE categories LIKE '%c3%' and categories LIKE '%c2%'; Would again get me rows 2 and 3, but not row 4
SELECT id FROM table WHERE categories LIKE '%c3%' or categories LIKE '%c2%'; Would again get me rows 2, 3, and 4
I don't like all the LIKE statements. I've found FIND_IN_SET() in the Oracle documentation but it doesn't seem to work in 10g. I get the following error:
ORA-00904: "FIND_IN_SET": invalid identifier
00904. 00000 - "%s: invalid identifier"
when running this query: SELECT id FROM table WHERE FIND_IN_SET('c2', categories); (example from the docs) or this query: SELECT id FROM table WHERE FIND_IN_SET('c2', categories) <> 0; (example from Google)
I would expect it to return rows 2 and 3.
Is there a better way to write these queries instead of using a ton of LIKE statements?
You can, using LIKE. You don't want to match for partial values, so you'll have to include the commas in your search. That also means that you'll have to provide an extra comma to search for values at the beginning or end of your text:
select
*
from
YourTable
where
',' || CommaSeparatedValueColumn || ',' LIKE '%,SearchValue,%'
But this query will be slow, as will all queries using LIKE, especially with a leading wildcard.
And there's always a risk. If there are spaces around the values, or values can contain commas themselves in which case they are surrounded by quotes (like in csv files), this query won't work and you'll have to add even more logic, slowing down your query even more.
A better solution would be to add a child table for these categories. Or rather even a separate table for the catagories, and a table that cross links them to YourTable.
You can write a PIPELINED table function which return a 1 column table. Each row is a value from the comma separated string. Use something like this to pop a string from the list and put it as a row into the table:
PIPE ROW(ltrim(rtrim(substr(l_list, 1, l_idx - 1),' '),' '));
Usage:
SELECT * FROM MyTable
WHERE 'c2' IN TABLE(Util_Pkg.split_string(categories));
See more here: Oracle docs
Yes and No...
"Yes":
Normalize the data (strongly recommended) - i.e. split the categorie column so that you have each categorie in a separate... then you can just query it in a normal faschion...
"No":
As long as you keep this "pseudo-structure" there will be several issues (performance and others) and you will have to do something similar to:
SELECT * FROM MyTable WHERE categories LIKE 'c2,%' OR categories = 'c2' OR categories LIKE '%,c2,%' OR categories LIKE '%,c2'
IF you absolutely must you could define a function which is named FIND_IN_SET like the following:
CREATE OR REPLACE Function FIND_IN_SET
( vSET IN varchar2, vToFind IN VARCHAR2 )
RETURN number
IS
rRESULT number;
BEGIN
rRESULT := -1;
SELECT COUNT(*) INTO rRESULT FROM DUAL WHERE vSET LIKE ( vToFine || ',%' ) OR vSET = vToFind OR vSET LIKE ('%,' || vToFind || ',%') OR vSET LIKE ('%,' || vToFind);
RETURN rRESULT;
END;
You can then use that function like:
SELECT * FROM MyTable WHERE FIND_IN_SET (categories, 'c2' ) > 0;
For the sake of future searchers, don't forget the regular expression way:
with tbl as (
select 1 ID, 'c1' CATEGORIES from dual
union
select 2 ID, 'c2,c3' CATEGORIES from dual
union
select 3 ID, 'c3,c2' CATEGORIES from dual
union
select 4 ID, 'c3' CATEGORIES from dual
union
select 5 ID, 'c4,c8,c5,c100' CATEGORIES from dual
)
select *
from tbl
where regexp_like(CATEGORIES, '(^|\W)c3(\W|$)');
ID CATEGORIES
---------- -------------
2 c2,c3
3 c3,c2
4 c3
This matches on a word boundary, so even if the comma was followed by a space it would still work. If you want to be more strict and match only where a comma separates values, replace the '\W' with a comma. At any rate, read the regular expression as:
match a group of either the beginning of the line or a word boundary, followed by the target search value, followed by a group of either a word boundary or the end of the line.
As long as the comma-delimited list is 512 characters or less, you can also use a regular expression in this instance (Oracle's regular expression functions, e.g., REGEXP_LIKE(), are limited to 512 characters):
SELECT id, categories
FROM mytable
WHERE REGEXP_LIKE('c2', '^(' || REPLACE(categories, ',', '|') || ')$', 'i');
In the above I'm replacing the commas with the regular expression alternation operator |. If your list of delimited values is already |-delimited, so much the better.
I have to resort to raw SQL where the ORM is falling short (using Django 1.7). The problem is that most of the queries end up being 80-90% similar. I cannot figure out a robust & secure way to build queries without violating re-usability.
Is string concatenation the only way out, i.e. build parameter-less query strings using if-else conditions, then safely include the parameters using prepared statements (to avoid SQL injection). I want to follow a simple approach for templating SQL for my project instead of re-inventing a mini ORM.
For example, consider this query:
SELECT id, name, team, rank_score
FROM
( SELECT id, name, team
ROW_NUMBER() OVER (PARTITION BY team
ORDER BY count_score DESC) AS rank_score
FROM
(SELECT id, name, team
COUNT(score) AS count_score
FROM people
INNER JOIN scores on (scores.people_id = people.id)
GROUP BY id, name, team
) AS count_table
) AS rank_table
WHERE rank_score < 3
How can I:
a) add optional WHERE constraint on people or
b) change INNER JOIN to LEFT OUTER or
c) change COUNT to SUM or
d) completely skip the OVER / PARTITION clause?
Better query
Fix the syntax, simplify and clarify:
SELECT *
FROM (
SELECT p.person_id, p.name, p.team, sum(s.score)::int AS score
, rank() OVER (PARTITION BY p.team ORDER BY sum(s.score) DESC)::int AS rnk
FROM person p
JOIN score s USING (person_id)
GROUP BY 1
) sub
WHERE rnk < 3;
Building on my updated table layout. See fiddle below.
You do not need the additional subquery. Window functions are executed after aggregate functions, so you can nest it like demonstrated.
While talking about "rank", you probably want to use rank(), not row_number().
Assuming people.people_id is the PK, you can simplify the GROUP BY clause.
Be sure to table-qualify all column names that might be ambiguous.
PL/pgSQL function
I would write a PL/pgSQL function that takes parameters for your variable parts. Implementing a - c of your points. d is unclear, leaving that for you to add.
CREATE TABLE person (
person_id serial PRIMARY KEY
, name text NOT NULL
, team text
);
CREATE TABLE score (
score_id serial PRIMARY KEY
, person_id int NOT NULL REFERENCES person
, score int NOT NULL
);
-- dummy values
WITH ins AS (
INSERT INTO person(name, team)
SELECT 'Jon Doe ' || p, t
FROM generate_series(1,20) p -- 20 guys x
, unnest ('{team1,team2,team3}'::text[]) t -- 3 teams
RETURNING person_id
)
INSERT INTO score(person_id, score)
SELECT i.person_id, (random() * 100)::int
FROM ins i, generate_series(1,5) g; -- 5 scores each
Function:
CREATE OR REPLACE FUNCTION f_demo(_agg text DEFAULT 'sum'
, _left_join bool DEFAULT false
, _where_name text DEFAULT null)
RETURNS TABLE(person_id int, name text, team text, score numeric, rnk bigint)
LANGUAGE plpgsql AS
$func$
DECLARE
_agg_op CONSTANT text[] := '{count, sum, avg}'; -- allowed agg functions
_sql text;
BEGIN
-- assert --
IF _agg ILIKE ANY (_agg_op) THEN
-- all good
ELSE
RAISE EXCEPTION '_agg must be one of %', _agg_op;
END IF;
-- query --
_sql := format('
SELECT *
FROM (
SELECT p.person_id, p.name, p.team, %1$s(s.score)::numeric AS score
, rank() OVER (PARTITION BY p.team ORDER BY %1$s(s.score) DESC) AS rnk
FROM person p
%2$s score s USING (person_id)
%3$s
GROUP BY 1
) sub
WHERE rnk < 3
ORDER BY team, rnk'
, _agg -- %1$s
, CASE WHEN _left_join THEN 'LEFT JOIN' ELSE 'JOIN' END -- %2$s
, CASE WHEN _where_name <> '' THEN 'WHERE p.name LIKE $1' ELSE '' END -- %3$s
);
-- debug -- inspect query first
-- RAISE NOTICE '%', _sql;
-- execute -- unquote when tested ok
RETURN QUERY EXECUTE _sql
USING _where_name; -- $1
END
$func$;
Call:
SELECT * FROM f_demo();
SELECT * FROM f_demo('sum', TRUE, '%2');
SELECT * FROM f_demo('avg', FALSE);
SELECT * FROM f_demo(_where_name := '%1_'); -- named param
fiddle
Old sqlfiddle
You need a firm understanding of PL/pgSQL. Else, there is too much to explain. You'll find related answers here on SO under plpgsql for every detail in the answer.
All parameters are treated safely, no SQL injection possible. See:
Define table and column names as arguments in a plpgsql function?
Table name as a PostgreSQL function parameter
Note in particular, how a WHERE clause is added conditionally when _where_name is passed, with the positional parameter $1 in the query sting. The value is passed to EXECUTE as value with the USING clause. No type conversion, no escaping, no chance for SQL injection. Examples:
Row expansion via "*" is not supported here
SQL state: 42601 syntax error at or near "11"
Refactor a PL/pgSQL function to return the output of various SELECT queries
Use DEFAULT values for function parameters, so you are free to provide any or none. More:
Functions with variable number of input parameters
The manual on calling functions
The function format() is instrumental for building complex dynamic SQL strings in a safe and clean fashion.
I have posted the similar question before but the solution to this question seems like it will be completely different, thus I hope this does not qualify for a repost.
Req:
I have 2 columns in a table named SETUPS with the following columns:
ID INTEGER NOT NULL
RPT_SCRIPT CLOB NOT NULL
RPT_SCRIPT has select statements in each record. Below is a statement in the clob column WHERE ID = 1:
SELECT ID,
Title,
Desc,
Type,
LVL_CNT,
TYPE_10 VALUE_10,
TYPE_9 VALUE_9,
TYPE_8 VALUE_8,
TYPE_7 VALUE_7,
TYPE_6 VALUE_6,
TYPE_5 VALUE_5,
TYPE_4 VALUE_4,
TYPE_3 VALUE_3,
TYPE_2 VALUE_2,
TYPE_1 VALUE_1
FROM SCHEMA.TABLE
WHERE ID = 1;
Currently I am writing these select statements manually for all records.
SETUPS.ID is mapped to another master table META.ID in order to build the select statement.
The column names with pattern TYPE_%, i.e. TYPE_1, come from the META table; there are total of 20 columns in the table with this pattern but in this example I've used only 10 because META.LVL_CNT = 10. Similarly if META.LVL_CNT = 5 then only select columns TYPE_1,TYPE_2,TYPE_3,TYPE_4,TYPE_5.
The column aliases, i.e. VALUE_1, are values which come from the corresponding column where META.ID = 1 (as in this example).
ID will always be provided, so it can be used to query table META.
EDIT
The column aliases which come from META table will never have a pattern as I have shown in my example, but with LVL_CNT, at runtime we will know the number of columns. I tried to #Asfakul's provided logic and built a dynamic sql using the column names retrieved dynamically but when using EXECUTE IMMEDIATE INTO I realized I don't know how many columns will be retrieved and therefore wouldn't be able to dynamically generate the alias name with this method.
Need an approach to automatically build this select statment using above information.. how can I achieve this? Please provide any examples.
You can use this as the basis
declare
upper_level number;
t_sql varchar2(1000);
l_sql varchar2(1000);
begin
select lvl_cnt into upper_level from
SETUPS S,META S
where s.id=m.id
l_sql:='SELECT ID,
Title,
Desc,
Type,'||
upper_level
for lvl in 1..upper_level
loop
t_sql:=t_sql||'type_'||lvl||','
end loop;
l_sql:=l_sql||t_sql
l_sql:=rtrim(l_sql,',');
l_sql:=l_sql||' FROM SCHEMA.TABLE
WHERE ID = 1;';
end
I recommend this approach, if you already know how to build dynamic SQL, then use this concept to build your query:
SELECT 'TYPE_' || LEVEL
FROM DUAL
CONNECT BY LEVEL <= 10 --10 could be a variable
How do I create a temporary result set for use in an SQL without creating a table and inserting the data?
Example: I have a list of, say 10 codes for example. I want to put this into a query, and then query the database to see which codes in this temporary list do not exist in a table.
If it was already in a table, I could do something like:
SELECT
ITEM_CODE
FROM
TEMP_ITEMS
MINUS
SELECT
ITEM_CODE
FROM
M_ITEMS
Is their a way without using PL/SQL, and pure SQL to create a temporary rowset before querying?
Please don't answer with something like:
SELECT 1 FROM DUAL
UNION ALL
SELECT 2 FROM DUAL
I am sort of thinking of something where I can provide my codes in an IN statement, and it turns that into rows for use in a later query.
Edit: so everyone knows my objective here, basically I sometimes get a list of product codes that I need to find which ones in the list are not setup in our system. I want a quick way to throw this into an SQL statement so I can see which ones are not in the system (rather than importing data etc). I usually put these into excel, then do a formula such as :
="'"&A1&"',"
So that I can create my comma separated list.
If you are using oracle 11g you can do this
with t as
(
select (column_value).getnumberval() Codes from xmltable('1,2,3,4,5')
)
SELECT * FROM t
WHERE NOT EXISTS (SELECT 1 FROM M_ITEMS M WHERE codes = M.ITEM_CODE);
or
with t as
(
select (column_value).getstringval() Codes from xmltable('"A","B","C"')
)
SELECT * FROM t
WHERE NOT EXISTS (SELECT 1 FROM M_ITEMS M WHERE codes = M.ITEM_CODE);
I would go with:
with t as (
select 1 as val from dual union all
select 2 as val from dual
)
select . . .
And then use "t" or whatever you call it, in the subsequent query block.
I'm not sure what the objection is to using the select method . . . just pop the values you want in a column in Excel and produce the code for each value by copying down the formula. Then paste the results back into your query interface.
If you want to use a temporary table, you can use the values clause. Alternatively, you can use string functions if you only want IN functionality. Put the values in a comma separated list and check to see if it matches a particular value:
where ','||<list>||',' like '%,'||col||',%'
This one is interesting because it's not a union and fit in a single select. You have to enter the string with delimiters ('a/b/c/def') two times though:
SELECT regexp_substr('a/b/c/def', '[^/]+', 1, ROWNUM) var,
regexp_substr('2/432/sd/fsd', '[^/]+', 1, ROWNUM) var2
FROM dual
CONNECT BY LEVEL <= length(regexp_replace('a/b/c/def', '[^/]', '')) + 1;
var var2
=== ====
a 2
b 432
c sd
def fsd
Note: Credits go to : https://stackoverflow.com/a/1381495/463056
So using the with clause it would give someting like :
with tempo as (
SELECT regexp_substr('a/b/c/def', '[^/]+', 1, ROWNUM) var,
regexp_substr('2/432/sd/fsd', '[^/]+', 1, ROWNUM) var2
FROM dual
CONNECT BY LEVEL <= length(regexp_replace('a/b/c/def', '[^/]', '')) + 1
)
select ...
or you can use it in a from clause :
select ...
from (
SELECT regexp_substr('a/b/c/def', '[^/]+', 1, ROWNUM) var,
regexp_substr('2/432/sd/fsd', '[^/]+', 1, ROWNUM) var2
FROM dual
CONNECT BY LEVEL <= length(regexp_replace('a/b/c/def', '[^/]', '')) + 1
) tempo
There are two approaches I would lean towards:
1. Global Temporary Table
Although you say you don't want to create a table, it depends on why you don't want a table. If you choose to create a Global Temporary table, the rows are only visible to the session that inserted them, so it's like having a private in-memory table but gives you all the benefits of a real table - i.e. being able to query and join to it.
2. Pipelined function
You can create a function that returns the results in a form that can be queried using the TABLE() operator. More info here: http://www.oracle-base.com/articles/misc/pipelined-table-functions.php
It's a bit hokey-looking. But you can parse a string into separate rows using regular expressions assuming you are using 10g or later. For example
SQL> ed
Wrote file afiedt.buf
1 SELECT REGEXP_SUBSTR('a,b,c,def,g', '[^ |,]+', 1, LEVEL) parsed_str
2 FROM dual
3* CONNECT BY LEVEL <= REGEXP_COUNT('a,b,c,def,g', '[^ |,]+')
SQL> /
PARSED_STR
--------------------------------------------
a
b
c
def
g
Personally, I would find a pipelined table function or a PL/SQL block that generates a collection easier to understand, but if you have to do it in SQL you can.
Based on your edit, if you are getting a list of product codes that is already in some sort of file, it would seem to make more sense to use an external table to expose the file as a table or to use SQL*Loader to load the data into a table (temporary or permanent) that you can query. Barring either of those options, if you really want to manipulate the list in Excel first, it would make more sense to generate an IN list in Excel and just copy and past that into your query. Generating a comma-separated list of codes in Excel only to parse that list into it's constituent elements in SQL seems like way too many steps.